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ABSTRACT 



Sfi, D A, EY KN0X libr ary 



U.S. Navy Submarine Approach Officers (AO) use tactical skills acquired from 
years at sea and a seemingly natural panache to accurately determine an enemy 
submarine’s bearing, range, course, and speed. This thesis investigates the effects of AO 
demographic differences and combat system employment methodologies on the ability to 
develop a timely and accurate firing solution. Employing a low-resolution submarine 
combat simulator, approximately 10 percent of the total pool of AO’s were taken through 
two of four pre-scripted initial contact scenarios. The AO’s were instructed to execute 
each scenario until they perceive that an accurate solution is obtained on the enemy 
submarine. The demographic differences of geographic location, ship type, and duty type 
are the top three traits that distinguish between success and failure. Further data analysis 
reveals differences in the information utilization of the simulator between successful and 
unsuccessful operators. Additionally, utilizing a survey administered to the subjects and 
basic display design principles, a notional command workstation for the next generation 
of submarine is developed. The conclusions of this research provide insight into the 
cognitive modeling, training, and selection of AO’s, as well as adding to the growing 
body of work in the design of military decision support systems. 
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DISCLAIMER 



The views expressed in this thesis are those of the author and do not reflect the 
official policy or position of the Department of Defense or the U.S. Government. 

The reader is cautioned that the computer programs developed in this research 
may not have been exercised for all cases of interest. While every effort has been made, 
within the time available, to ensure that the programs are free of computational and 
logical errors, they cannot be considered validated. Any application of these programs 
without additional verification is at the risk of the user. 

The simulator utilized in this research was developed by the Naval UnderSea 
Warfare Center (NUWC). Any inquires concerning its origination or use should be 



directed to NUWC. 
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EXECUTIVE SUMMARY 



Recent changes in U.S. Naval doctrine have underscored the need for Navy ships 
to operate in the littoral regions of the world. Operating in proximity to congested coastal 
regions present unique challenges to the military decision-maker. Frequently, there is no 
clear-cut answer about a specific situation or decision. Rapidly unfolding events result in 
severe time pressure and extreme, often fatal, consequences. Current real-time battle- 
space management systems are well suited to the demands of a protracted blue water 
conflict. However, their ability to be optimally employed in the emergent fast-moving 
littoral environment is questionable. 

Project NEMO (Naval Expertise and Modeling) was initiated by the Naval 
UnderSea Warfare Center (NUWC) to explore Combat Control System (CCS) 
modifications to best meet Submarine Approach Officers (AO) needs in these new 
environments. Project NEMO’s ultimate goal is “to provide for the layout, design, and 
implementation of a Submarine Command Workstation (CWS) to be incorporated into 
the New Attack Submarine (NSSN)”. Utilizing a small cadre of submarine officers 
stationed at NUWC, Project NEMO researchers are currently in the process of modeling 
the AO's cognitive processes during tactical engagements. While the initial CCS for the 
NSSN is in the final stages of development, it is postulated that lessons learned from 
Project NEMO will influence future upgrades and provide for system improvements. 

This thesis has three primary objectives. The first objective is to explore possible 
relationships between both subject demographic differences and CCS employment 
strategies in the prediction of a successful engagement. The second objective is to 
provide data in the development of a notional "AO friendly" CWS interface. It is 
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postulated that this interface will allow the AO to more accurately evaluate relevant 
tactical information with less effort through the use of more appropriate information 
presentation and display formatting. The third objective is to provide for principles 
which could be applied in the areas of personnel selection, qualification, and training. 

A low-resolution computer-based CCS simulation was used to run subjects 
though two of four pre-scripted initial contact scenarios in a modified Latin square 
design. The subject pool comprised approximately 10% of the active duty U.S. AO’s. 
The study used two objective measures of effectiveness, solution accuracy at time of fire 
(TOF) and weapon to target closest point of approach (CPA), and one subjective measure 
of effectiveness; the determination of a “successful” engagement. In addition, subjects 
participated in a design survey to provide feedback on not only the simulation, but also 
their desires for the next generation CWS. 

An exploratory statistical analysis was performed on the subject-generated 
database with the goal of determining if demographic distinctions and CCS usage 
strategies among AO’s could be used to predict the quality of a simulated engagement. 
Regression techniques were then applied to the MOE’s obtained for the subjects. 
Geographical location, ship type, and duty type were the three most important 
determinants in the prediction of a successful engagement. Surprisingly, no distinction 
between experts and novice operators was found with the MOE's. Coupling this finding 
with an almost zero correlation between accuracy and number of years at sea, a hallmark 
for expert designation, raises the question as to whether a good AO is "bom or made". 
Analysis of the linear relationships between TOF and CPA revealed two important 
observations: 1) the shorter the engagement, the more accurate target solution, and 2) 
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expert vs. novice distinction is negligible, while geographical location, ship type, and 
education are significant factors in predicting success. 

A single step link analysis was examined on both display usage and solution 
generation. Rather than examining the differences between expert and novice AO’s, the 
differences between successful and unsuccessful AO’s were used instead. It was shown 
that successful and unsuccessful subjects do in fact demonstrate variance in the cognitive 
progression of links in both display usage and solution generation. The successful 
subjects were shown to follow standard submarine liturgy and submerged approach 
methodology while unsuccessful subjects were observed to be acting in an ad hoc manner 
(i.e. "easter egging”). This further reinforces the benefit of reliance on established 
methodologies and techniques and provides for applications in the periodic reevaluation 
and training of AO's. 

Results from the two link analyses were contrasted with both subject evaluations 
of the current CCS’s and subject desires for CCS attributes into a preliminary CWS 
design. While originally it was envisioned subjects would only "brainstorm" the CWS 
fire control module, subject dissatisfaction with current CCS's and the desire for a 
revolutionary leap in its employment, significantly expanded the scope of subject inputs. 
The final product can be viewed as a notional design concept for a complete CWS. 

The main value of this study is the continued exploration of the cognitive 
processes of the submarine AO. The results obtained have implications in initial 
selection, periodic reevaluation, career progression, and training regimen of personnel. 
The data collected in the course of this research will continue to provide additional 
insight and future work should continue to explore the hypothesis not yet examined. 
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I. INTRODUCTION 



Changes in United States strategic priorities have generated an increased need for 
U.S. Navy ships to operate in the littoral regions of the world (Department of the Navy, 
1996). Naval doctrine specifies that the near term battle-space will be exclusively in the 
littorals. Littoral regions, defined as the area within 200 nautical miles of the coastline, 
necessitate operations in the congested, confined water and airspace close to an 
adversary’s shores (Hutchins, 1995). Littoral warfare presents unique challenges to the 
decision-maker for it entails multi-faceted scenarios characterized by rapidly unfolding 
events involving contact identification, comprehension of intent, consideration of 
available responses, and forecasting potential consequences. Frequently, there are no 
clear-cut answers about specific situations and often these events can have extreme, even 
fatal, consequences for errors (Collyer & Malecki, 1998). Current real-time battle-space 
management systems are well suited to the demands of protracted blue water conflicts; 
however, their viability in the littoral environment is questionable. 

A. BACKGROUND 

There have been several well-publicized and unfortunate events that have focused 
the military’s attention on the difficult decisions facing U.S. Naval commanders in the 
littorals. The 1987 USS STARK (Levinson, 1997) and the 1988 USS VINCENNES 
(Rogers, 1996) incidents are still fresh in the Navy’s collective consciousness. In the 
case of the STARK, a harassing Iraqi F-4 jet was allowed to shoot two anti-ship missiles, 
severely damaging the ship and tragically resulting in 37 crew fatalities. This situation 
was predicated by the confusing and contradictory rules of engagement (ROE) under 
which the STARK was operating. In the case of the VINCENNES, an Iranian 
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commercial jetliner carrying 288 civilians was mistakenly identified as an inbound 
hostile aircraft, and was consequently shot down. The official after action report on the 
VINCENNES incident found that the decision making process was influenced by the 
events surrounding the STARK the year before. While circumstances leading up to, and 
surrounding the two events are very different, in both cases the basic mistake remains the 
same. The Commanding Officer (CO), operating under conditions of extreme stress, 
made the wrong decision about whether or not to engage an inbound aircraft. 

In recognition of the complex and difficult decisions that led up to the STARK 
and VINCENNES incidents, the Tactical Decision-Making Under Stress (TADMUS) 
program was initiated (See Appendix A for a full discussion of TADMUS). The 
objective of the TADMUS program was to explore the areas of human factors and 
training technology in order to develop and apply principles that could help avoid such 
incidents in the future (Bowers & Salas, 1998). TADMUS strives to act as an intelligent 
assistant to the CO/Tactical Action Officer (TAO) team of an AEGIS cruiser during 
periods when the mental workload exceeds the teams combined cognitive capacity. It 
enhances the visual stimulus presented to the team while simultaneously overcoming 
their cognitive limitations in support of a recognition primed heuristic for the current 
tactical scenario. Furthermore, it attempts to model the human cognitive decision making 
process by independently forming hypotheses concerning target motivation and 
subsequent own ship response options. At the time of this research, TADMUS was in the 
process of undergoing final approval prior to implementation aboard AEGIS cruisers 
(Morrison, Kelly, Moore, & Hutchins, 1998). 
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There are several “submerged events" similar to the STARK and VINCENNES 
incidents that currently weigh heavily on the U.S. Navy Submarine community. The 
classification of this paper does not permit a frank discussion of such incidents, but it 
suffices to say that such incidents do occur (Bowermaster, 1993; Clancy, 1997). In fact, 
only recently has the 1993 underwater collision between the USS GRAYLING and a 
Soviet Delta class SSBN in the North Sea been made public. The Submarine community 
has realized the potential benefits of a TADMUS-like system and is eagerly pursuing the 
acquisition of such a system. 

The Naval UnderSea Warfare Center (NUWC) initiated Project NEMO (Naval 
Expertise and Modeling) in the mid-1990's to examine possible changes for the current 
Combat Control System (CCS) that best meet the needs of the Submarine Approach 
Officers (AO) during a tactical engagement. Project NEMO’s ultimate goal is “to 
provide for the layout, design, and implementation of a Submarine Command 
Workstation (CWS) to be incorporated into the New Attack Submarine (NSSN)” 
(Kirschenbaum, 1997). Project NEMO researchers are currently modeling the AO's 
cognitive processes during tactical engagements based on a small a small cadre of 
Submarine officers stationed at NUWC (Kirschenbaum, 1997). However, the program is 
still in the infancy stage, with many years of difficult work ahead. While the initial CCS 
for the NSSN is currently in its final stages of development, it is hoped that Project 
NEMO's findings will influence future upgrades and provide for substantial 
improvements. 
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B. OBJECTIVE STATEMENT 



By understanding and capitalizing upon the AO’s cognitive processes, the CWS 
could allow for a more accurate evaluation of relevant tactical information with less 
effort as compared to existing tactical displays. Thus, this thesis has two primary 
objectives: 

1) To assess the relationship between both AO demographic variables, such as 
rank, experience, education, etc., and AO CCS employment strategies to the 
probability of a "successful" tactical engagement. These variables and 
strategies could then be used as weighting factors in the areas of personnel 
selection and qualification. 

2) To develop a basic design concept for an "AO friendly" CWS interface based 
on the CCS informational usage, display employment frequency, and AO 
subjective inputs. This interface would be the centerpiece for an AO decision 
support system. This system could also be used as a training aid during 
mission preparation. 

C. STATEMENT OF THE PROBLEM 

Cognitive task analysis of the AO and their goal allocation structure has been 
examined in detail as an initial part of Project NEMO (Kirschenbaum, 1997). The next 
stage focuses on what accounts for the differences between "successful" and 
"unsuccessful" AO’s. Specifically, the questions to be answered are: are the mental 
models of AO’s, as they relate to informational usage from the CCS, the same and do 
successful and unsuccessful AO’s share the same level of situational awareness 
throughout the course of a tactical scenario? It is thought that the AO's CCS usage 
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strategy and success rate might be related to their status as either an "Expert" or "Novice" 
AO. Demographic variables such as current job, ship type, location stationed, general 
computer experience, combat system experience, and post-secondary education might be 
also be factors that contribute in the distinction between CCS usage strategies and the 
prediction of success. 

Currently, AO’s have no display specifically designed to give them the “big 
picture” of an engagement. Figure 1.1 shows the typical tactical aid that submarine AO’s 
use to plan and execute attacks. The tactical aid is actually an 8-inch square piece of 
plexi-glass covering a standard naval maneuvering board. The writing on the tactical aid 
comes from a black grease pencil used by the AO. Information such as own-ship’s 
current and past course, speed, and depth, current sub-surface and surface environmental 
conditions, ancillary contact parameters, as well as pertinent contact kinematics are 
typically detailed on the tactical aid. It should be noted though, that there is neither a 
defined set of requirements for what information that should be placed on the tactical aid 
nor a prescribed format for that information. It is simply a matter of personal choice. 
Additionally, in order to construct this tactical aid, the AO compiles information from up 
to fourteen distinct displays, five individuals, and three separate locations within the 
submarine control area. Once gathered, the data is manually correlated for each target in 
the current situation, and then all targets are correlated to achieve the big picture. 
Compounding this situation, the submarine control area typically has 30-45 men in a very 
confined space of as little as 25 x 25 feet. Such environmental distracters only serve to 
degrade a commander’s already reduced capability to successfully execute his primary 
task. This is an unacceptable situation, and therefore an attempt will be made to develop 
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a notional submarine command level display concept that provides for more complete 



utilization of information and an increased situational awareness (Kirschenbaum, 1997). 




Figure 1.1 Typical AO Tactical Aid. 



D. SCOPE, LIMITATIONS, AND ASSUMPTIONS 

The present effort is confined to the evaluation of active duty U.S. Naval 
Submarine Officers in the use of a simulated CCS. Due to the simulator's low resolution, 
limits imposed on scenario run time, subject workload concerns, and military 
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classification, some realistic elements were impossible to incorporate. All subjects are 
assumed to be willing participants whose motivation is driven by pride and 
professionalism. 

This paper is organized into five sections. First, the motivation for and objective 
of this research is described. Next, an in depth review of the literature supporting the 
objectives is presented. The methodology, which comprises both the experiment 
configuration and outcome generation, is described in the third section. The fourth 
section contains the results of exploratory analysis performed on the subject-generated 
database. The final section discusses the results and provides lessons learned and 
recommendations for follow on work. 
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n. REVIEW OF THE LITERATURE 



Edwards (1988) defines Human Factors (HF) as the “technology concerned to 
optimize the relationships between people and their activities by the systematic 
application of the human sciences, integrated within the framework of human 
engineering.” It attempts to integrate complex human performance concepts with 
statistical and stochastic methods to adequately capture the "man-machine" relationship. 
Because of the ambiguity involved, the study of HF has not gained universal acceptance. 
However, its benefits and contributions over the last 50 years cannot be disputed 
(Lederer, 1988). To provide a common core of HF terminology and concepts to the 
reader, a brief overview of Human Information Processing, Naturalistic Decision- 
Making, Situational Awareness, Schema Theory, Expert vs. Novice Performance, 
Display Design, and Link Analysis Techniques is presented. The military applicability of 
these areas is then used to address the core issues of this research. 

A. HUMAN FACTORS CONCEPTUAL OVERVIEW 

1. Human Information Processing 

Human Information Processing (HIP) is the vehicle by which people decipher 
data. A principle feature of HIP models is the assumption that a series of stages or 
mental operations occur between external stimuli and self-directed responses. Much HF 
research is directed at isolating, and then characterizing each of these stages. Attributes 
such as capacity, duration, and representational forms of the particular stage are studied. 
While the end result remains elusive, there are several generally accepted models with the 
most widely accepted one being the Wickens and Flech (1988) Attention Resource Model 
(see Figure 2.1). 
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Figure 2.1 Attention Resource Model (Wickens & Flech, 1988). 

The first stage of the Attention Resource Model, as depicted in Figure 2.1, is the 
Sensory Store. It is generally assumed that there is a separate sensory store for each 
sensory modality. In this store, the externally generated physical energy is transformed 
into internal neural energy and is represented in terms of its salient physical features. 
This information lasts only a brief time and does not require any attentional resources. 
While the first stage is relatively simple and uncomplicated, the second stage, Pattern 
Recall, is both the most crucial and least understood of all of the stages. It is at this stage 
that the physical stimulation in the sensory stores is integrated into meaningful elements. 
For example, the dark curved lines on this piece of paper are processed as meaningful 
letters and words. This process is very complex and is limited in its execution by the 
pool of available attentional resources. These patterns are typically stored in long term 
memory for iater recall. 
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The HP Attention Resource Model contains two distinct ways to store 
information: Working Memory, and Long-term Memory. Working memory represents 
the information currently being used by the information processor. The capacity of 
working memory is limited in both its scope and size. Long-term memory represents 
information available to the information processor, but not currently in use. Unlike 
working memory, long-term memory capacity is essentially unlimited. 

The third stage, Decision Selection, is where a stimulus is recognized and a 
decision must be made regarding that stimulus. Various options are available at this 
point: the information can be stored for use at some future date, it can be integrated with 
other available information, or it may initiate a response. Each of these options generates 
their own associated costs and benefits that must be considered when choosing a single 
course of action. The decision generated in this stage is finally passed onto the last stage, 
Response Execution. In this stage, the selected decision is translated into a series of 
unconscious and conscious motor actions and sequences. The resulting outcomes, by 
way of a feedback loop, then become input to the sensory stores, which in turn can be 
interpreted and entered as relevant data in selecting the next response. As with the second 
stage, these final two stages are limited in their scope by the available attentional 
resources (Kantowitz & Sorkin, 1983). 

2. Naturalistic Decision Making 

In the early 1990’s, there was a radical shift in the perception of human decision- 
making (Collyer & Malecki, 1998). The analysis of actual decision strategies that 
humans use in solving complex, real-world conditions was termed “Naturalistic Decision 
Making” (NDM) (Klein, 1993). “Naturalistic” psychologists assert that expert decision- 
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makers rarely use the traditional resource intensive strategies of classical decision- 
making in order to make decisions in the presence of complex, adverse, and time 
constrained scenarios. Rather, experts rely on internal heuristics formed by the very 
nature of being recognized system experts (Kozlowski, 1998). It is postulated that 
experts tend to recognize scenarios based on how closely the current situation matches 
one previously experienced (Kaempf & Militelo, 1992). The expert then recalls the 
previous actions taken in the stored scenario, evaluates those actions against the current 
situation, makes appropriate modifications within the current time constraints, and then 
activates the schema containing the actions. Rather than employing an optimization 
routine, the schema's activated typically pursue a satisficing strategy (Klein, 1993; 
Hutchins, 1995; Kirschenbaum, 1997). In other words, the expert selects a course of 
action which is not necessarily the best, but is still reasonable and will result in a outcome 
that is within the constraints of the situation at hand. 

There are two models of decision-making currently used in “naturalistic” settings, 
Recognition Primed Development (RPD) and Explanation Based Reasoning (EBR) 
(Klein, 1993). RPD, more commonly known as feature matching, “occurs when the 
decision maker recognized the features of the present situation as similar or identical to 
those of a previous situation” (Kaempf & Militelo, 1992). An unconscious correlation 
then triggers the recall of all pertinent information learned about that particular situation: 
clues, cues, goals, expectations, plans of action, and outcomes that were observed in the 
past. Therefore, it is thought, expert decision-makers rely on detailed heuristics stored in 
long term memory to control decision making in similar, but different situations 
(Hutchins, 1996). EBR, more commonly known as story generation, occurs when the 
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decision-makers find themselves in complex situations without all the required 
information, the information is self-contradictory, and/or certain correlating events are 
absent (Bowers, 1998). The decision-maker then develops “neural links”, or causal 
relationships, between facts to produce a working representation of the situation (Klein, 
1993). Once constructed, the new model of the situation is acted upon by the decision- 
maker within the current environmental conditions. 

Today, advanced Decision Support Systems (DSS) involved in such industries as 
nuclear power plants, off-shore oil platforms, and emergency service providers utilize a 
mixture of displays that support both RPD and EBR cognitive processing (Wickens, 
1992). These displays, respectively, are termed “status” and “command”. Status displays 
are characterized by their inclusion of “why” information and informs the user of what is 
known and assumed about the current situation. Status displays allow the user to 
examine differing hypotheses without committing to a specific course of action. This is 
in contrast to the “command” type display, which present information in list or bullet 
format and tells the user what to do without explaining the motivation behind the actions. 
This type of display is most often utilized in situations when quick responses are required 
of inexperienced or heavily tasked operators. The most common display type used for 
military decision support systems in the command display (Irizarry & Knapp, 1986). 
However, research indicates that by providing decision support in the form of a 
combination of command and status displays, an increase in the overall accuracy in the 
understanding of system and environmental states occurs (Kozlowski, 1998). 
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3. Situational Awareness 



Enhancement in the understanding of system and environmental states is referred 
to as Situational Awareness (SA). For all current accepted models of NDM, SA is the 
sum and substance: sizing up the situation, understanding a scenario, defining the 
problem, categorizing the circumstances, constructing a representation, making mental 
models, painting a picture, or creating images (Federico, 1995). Decision-making in 
realistic settings is a process of not only producing and updating a representation of the 
perceived situation, but also specifying and appraising possible actions. 

Unfortunately, perception, action appraisal, and mental choices are difficult to 
measure. However, since SA is the crux of user comparison, measures of effectiveness 
(MOE's) must be obtained (Federico, 1995). Typically in military settings, two 
approaches are taken in the measurement of SA. The first is the measurement of critical 
factors or preconditions that lead to, or inhibit SA. By the measurement of two 
secondary load factors, attention and working memory, the acquisition and interpretation 
of information from the environment necessary to form SA can be ascertained (Endsley, 
1995). The second is the attainment of an objective MOE. This is typically measured in 
terms the military mind easily understands: minimum time or range, hit or miss, or life or 
death. 

Of the current NDM models in use today, some link SA to the selection of action 
alternatives while others indicate that SA precedes and initiates the selection of action 
alternatives. Hammond’s (1986) Intuitive Decision, Lipshitz’s (1989) Matching Mode 
Decisions, and Rasmussen’s (1983) Skill-based and Rule-based behaviors are all 
examples in which the perceived situation directly determines the chosen action. 
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Alternatively, Beach’s (1990) Framing Action Selection Criteria, Connolly’s (1982) 
Cognitive Mapping, and Klein’s (1990) model of Critical Decision Making (CDM) are 
examples in which recognizing the situation precedes the determination of the chosen 
alternative. 

NDM models as a group tend to emphasize certain salient cognitive processes that 
are associated with SA. Representing problems using knowledge structures or schema 
(Connolly, 1982; Beach, 1990), categorizing situations (Rasmussen, 1983; Klein, 1989; 
Lipshitz, 1989), and using storytelling or mental modeling (Hammond, 1986; Lipshitz, 
1989; Beach, 1990) are some of the different methods espoused. However, the common 
thread running though all of these models is that the knowledge necessary to assess and 
recognize situations for NDM is incorporated into stored schema (Federico, 1995). 

4. Schema Theory 

Schemas are commonly defined as stored metal representations of previous 
experiences (Bailey, 1980). In other words, a schema is like an organized memory unit, 
similar to a stored computer program. These memory units can be stored and 
subsequently recalled using any of the five senses (Wickens, 1994). Once invoked, a 
schema is typically executed by a series of automatic and unconscious motor action 
sequences. For example, starting your car and driving to work, making toast for 
breakfast, or calling relatives on the telephone, are all examples of everyday tasks most 
people accomplish in an unconscious manner. An informal representation of the schema 
theory model is shown in Figure 2.2. 
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Concious Thought 



Unconcious Thought 




Figure 2.2 Schema Theory Model (NPGS School of Aviation Safety Handout). 

In the current context of NDM, it is logical to infer that it is the stored schema that 
drives decisions. Consequently, this schema-driven decision making is primarily a RPD 
process. Stated differently, decision-making is comprised of recognizing previous 
scenarios and situations, the courses of action taken, and the eventual outcomes (Zachary, 
Ryder & Hicinbothom, 1998). It is postulated that entire patterns and sequences of 
events are employed in their entirety to facilitate recognition and assessment of similar 
scenarios. When similar situations are recognized or experienced, stored schema is 
activated, and past successful actions are applied to the present situation. This discussion 
is predicated on the notion that the decision-maker in fact, possesses the necessary 
schema in the form of neural-networked identification, action, and solution links. This 
assumption of correct mental representation and its correct employment, is the primary 
distinguishing trait of the so-called expert decision-maker. (Gick & Holyoak, 1983; 
Beach, 1990; Wickens, 1992). 
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5. Expert vs. Novice Performance 

The core elements of expert vs. novice distinction and subsequent performance 
implications have been the subject of numerous investigations (Federico, 1995; Glaser, 
1985; Klein, 1993; Wickens, 1992). These observations have been drawn from the 
analysis of decision-makers that routinely operate under conditions of extreme time 
constraints and stress: emergency room doctors, aircraft pilots, nuclear plant operators, 
etc. When taken as a group, the observations of previous researchers seem to share 
several important aspects: 1) experts tend to excel primarily because they possess 
extensive knowledge in their respective areas, 2) experts distinguish patterns in task 
execution that indicate superior knowledge organization skills, 3) experts perform tasks 
quicker with less errors, 4) experts process superior STM and LTM, 5) experts perceive 
and categorize situations at a deeper and more principled level, 6) experts initiate and 
maintain correct mental representations of the situation and inherent constraints at hand, 
7) experts posses superior self-monitoring skills when checking for errors, 8) experts 
spend proportionally more time building the basic problem description than seeking 
solutions, 9) experts possess a larger set of stored schema, and 10) expert knowledge is 
structured, organized, and inter-linked to a greater degree (Glaser, 1985; Glaser & Chi, 
1988). 

Expert knowledge, however, has been shown to suffer from severe biases that 
inhibit proper decision-making. Experts as a group tend to have difficulty in responding 
to feedback and shifts in the nature of the situation after a course of action has been 
executed. Problems with misleading feedback, limited attention to delayed feedback, 
selective perception of feedback, and selective influence on outcome have been 
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conclusively shown in studies (Wickens, 1993). Novices, due to their lack of pre- 
conceived notions and institutional biases, tend to react better to ever changing 
environments. 

The results from these investigations of skilled behavior all demonstrate that 
fundamental differences exist between expert and novice decision-makers regarding 
knowledge structure, pattern perception, performance speed and accuracy, memory 
capacity, and problem perception. Once identified, these differences cannot only be 
capitalized upon, but if necessary, corrected through selection methods, training 
regiments, or system modifications (Howell, 1998). 

6. Display Design 

Since it is the man-machine interface that most users are concerned with, the 
question is now: How does the interface designer account for and exploit differences in 
the varying levels of user proficiency? Once the appropriate mental model of the user 
has been constructed, how does a designer go about the construction of a new 
interface/display? What principles do they rely on to ensure the user derives maximum 
benefit? In general, a display’s effectiveness is the functional combination of five 
specific and distinct attributes: Legibility, Readability, Accuracy, Compatibility, and 
Timeliness (Bailey, 1980). 

Legibility simply means that the display can be easily read. A detailed knowledge 
of typefaces, letter sizes, spacing, etc., all come into play. If the display is illegible, it 
will fail to communicate needed information to the user, and all of the other 
characteristics will become useless. Legibility, however is not an absolute property 
(Meister, 1965). Both the external aural and visual environment affects its use. 
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Readability implies that the display communicates well, and the user perceives, 
without effort, the meaning or purpose of what is being presented. This in turn implies 
either an equal level of expertise among users, or a display that can adapt or differentiate 
amongst users. Readability is also achieved in part by having the appropriate sequential 
order of presentation, or grouping of information, to the user (Ramsey & Atwood, 1979). 

Accuracy is a double-edged sword. On one hand, the display must be designed to 
elicit accurate information from the user. Instructions or headings that are vague, 
ambiguous, or misleading may result in inaccurate entries by the user. On the other hand, 
the display must transmit accurate information to the user (Wickens, 1992). Failure to do 
so might result in the formation of inaccurate hypothesis and the subsequent incorrect 
courses of action. 

A display must be compatible with the user’s knowledge and skill, other 
concurrent activities being performed, and again, the working environment in which it is 
being used. Compatibility is perhaps the most important factor in both performance and 
acceptance by the user. Experience has shown that sometimes systems fail not for 
technical reasons but for reasons of user acceptance (Engel & Eruda, 1975). The common 
sense remedy to ensure compatibility and cultivate user acceptance is to clearly establish 
user requirements early in the developmental process, and then ensure that the new 
displays adequately reflects these requirements. 

The final attribute in display design concerns the timeliness of the data built in 
and presented by the display. In simple terms, the display must present information 
available to the user while it is still useful. Processing or compiling (to use computer 
jargon) delays are to be avoided in any low-ordered system. Also, a well-designed 
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display should present only essential information in such a form as to require only a 
minimum amount of interpretation (Bailey, 1980). Information, just for information’s 
sake, has no place and only serves to confuse the operator. 

7. Link Analysis 

In addition to the five factors of good display design, two more issues should be 
addressed. They are the concepts of Grouping and Standardization. Grouping techniques 
are instrumental in helping to organize information and are comprised of four 
fundamental traits: Sequence, Frequency, Function, and Importance. Standardization 
techniques assure that information is presented in compatible sensory modes and that 
they do not conflict with any preconceived notions and/or biases held by the users. 
(Bailey, 1980). 

Link Analysis groupings all share the same core principle: items should be 
grouped together based on their relative relationship to each other (Bailey, 1980). This 
relationship can be defined in terms of sequence of use (i.e. left to right and top to 
bottom), frequency of use (i.e. bivariate normal distributions of item location in the 
display), functional allocation (i.e. changing the layout of the display based on operating 
mode required by the user), and importance placement (i.e. placing critical items in the 
best position on the display to insure that they are not overlooked). Each type of 
grouping lends itself to different data collection methods. Sequential relationships are 
identified by using a process called procedural flow analysis. Frequency analysis is 
accomplished via a process called link analysis. Functional and Importance relationships 
are most often determined by a combination of historical, procedural, and subject 
operating methods and modes (Chapanis, 1959). Each type of Grouping has its 
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contribution to make, and it is the designer’s task to undertake a deliberate process of 
elimination, weighting, and judgment to arrive at the “best” possible design. When 
conflicts arise, as they are wont to do, the designer must give the highest precedence to 
meeting human performance requirements above all others. 

Standardization itself is comprised of two types: standardization of items on a 
specific display and standardization of display format (Bailey, 1980). Standardization of 
items refers to the goal that all items in the system with the same information should have 
the same modality. This modality includes not only its name, structure and physical 
appearance, but also its encoding algorithm. Standardization of display format is the 
principle in which the interface designer attempts to place items that appear on multiple 
displays in the same geographical location on all displays. This intentional placement 
reduces the workload on users by allowing the user to capitalize on “location” memories 
(Davis, 1966). Although standardization is desirable, it should not take precedence over 
the grouping principles described earlier. 

As stated before, there are several avenues that facilitate proper human factor data 
collection techniques: Procedural Flow Analysis, Articulation Testing, Critical Incident 
Analysis, Time Line Analysis, and Link Analysis just to name a few. Each method is 
used to address similar, but very specific questions. The latter, link analysis is the 
observational technique for determining the relative association among system 
components or elements (Bailey, 1980). The results of link analysis are typically 
expressed in either frequencies or probabilities. The textbook application of link analysis 
is the “office desk arrangement problem” (Chapanis, 1959). In this situation, the office 
layout designer is concerned with arranging the desks of several employees in order to 
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facilitate the greatest communication with the minimum movement of employees. Data 
is collected on the number and distribution of interactions between employees and a 
seating arrangement then evolves. However, elementary applications of link analysis do 
not take into account issues of time partitioning, usefulness, or hierarchical associations 
among subjects. Therefore, link analysis is best used when determining the layout and 
arrangement of people and machines in systems, as well as the displays they employ. 

B. MILITARY APPLICABILITY 

1. Submarine Approach Methodology 

Regardless of a submarine’s operational tasking, successful units provide the CO 
with quick and accurate information on all contacts. The task of correlating and 
transforming the available raw data into the necessary picture is borne by the AO 
(Clancy, 1997). The AO performs the role of senior decision-maker and advisor to the 
CO during an encounter with a hostile target. 

Information provided to the AO is, unfortunately, cryptic and uncertain. The AO 
will never be able to confirm theories directly, but may only observe how closely reality 
conforms to them (Clancy, 1990). The AO hunts an enemy submarine that is most likely 
also hunting for him. Unlike other modem forms of combat, the AO has no visual 
contact with an opponent. The medium of the ocean inhibits visual utilization to only a 
few feet (Sharpe, 1998). Rather, those that operate under the sea utilize aural information 
exclusively. 

A useful analogy of the AO's task would be the hypothetical situation of two 
blindfolded prizefighters in a boxing ring. In the ring, each fighter seeks the other and 
each has a brutal power punch available to use if necessary. However, they are not alone 
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in the ring, as innocent non-combatants (referee, trainers, scorecard keepers, etc.) and 
even the occasional domestic animal wanders occasionally about. Fortunately, each 
knows the approximate layout of the ring, i.e. the location of the ropes, stools, and other 
distinguishing traits. During his search, each relies on his own aural abilities to identify 
other wanderers, decide whether they might be the opponent, and attack if within range of 
a punch. Still, simply detecting the enemy does not imply the ability to hit him. First the 
fighter must roughly decide the distance of the enemy, using the level, pitch, and 
directivity of the noise. Once the enemy range is identified, the fighter must approach his 
opponent, within effective striking range, remembering that the chances of being counter- 
detected increases each second. Using similar, but somewhat more complex methods, the 
fighter also determines the enemy’s moving direction and speed. Once the fighter is 
satisfied that the enemy’s "solution" has been determined, a punch is thrown, and then the 
fighter quickly retreats to avoid the enemy’s counter-punch, which will assuredly be 
thrown in response. A successful engagement will find the fighter safe, his opponent 
dispatched by a wicked right cross, and no enemy counter-punch contacting the fighter 
due to some suicidal last effort. 

Even the forgoing discussion simplifies the role of the AO considerably. Physics 
presents difficult challenges in a real scenario; e.g. sound does not travel in straight lines, 
rather it bends and bounces as it travels through the ever-changing ocean medium 
(Sharpe, 1998). Received sound has often been reflected off the bottom or surface of the 
ocean, so the source lies not on a line but instead a hyperbola opening behind, or in front 
of, the sensor. The phenomena of Convergence Zones, Ducting, Shadow Zones, Arrival 
Paths, and Limiting Rays, just to name a few, even further mire the situation. This 
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discussion is only intended to give some idea of the various tasks and concerns the AO 
faces during a submerged approach. 

2. Previous Research 

Due to the inherently secretive nature of submarine design and construction, there 
is a noticeable absence of unclassified human factor analysis in the submarine 
community. Previous work has shown that naval officers engaged in Anti-Submarine 
Warfare (ASW) and Anti-Surface Warfare (ASUW) show a pre-disposition to data 
representation in analogical, or geo-situational, frames of reference (Kirschenbaum, 
1995; Hooper, 1994). In addition, it has been shown that while senior ASW decision- 
makers strive for optimality in their endeavors, certain inherent and specific cognitive 
limitations and biases prevent this: they include 1) working memory limitations, 2) a bias 
toward overestimation of enemy submarine maneuverability, and 3) a bias toward 
overestimation of sonar accuracy (Wohl, 1989). 

There is a wealth of research on human factor analysis concerning aviation issues, 
and to a lesser extent, military decision-makers in general. Several texts (Wickens, 1992, 
Wiener & Nagel, 1988, Pruitt, 1982) give excellent examples of the depth and breadth of 
HF aviator work. The cognitive link between aviators, surface warfare officers, and 
submarine AO’s was established by Hooper (1994), and is used as an anchor for the 
principles applied in the course of this research. It is in fact reasonable to assume that, in 
general, military decision-makers all share the same cognitive abilities and limitations. 

3. Current Research 

Unlike the multitude of research projects underway in aviation areas, the only 
major unclassified submarine related human factor analysis project currently under works 
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is Project NEMO (Kirschenbaum, 1997). Early work in project NEMO has concentrated 
on examining the goal and sub-goal structure of a small cadre of senior submarine AO’s. 
Preliminary results indicate a very shallow goal structure, as well as a bias toward 
analogical schema orientation. Secondary results indicate a possible limit on both the 
short term and long term working memory of AO’s that must be acknowledged and 
exploited. Current work surrounds expert vs. novice issues as well as preliminary display 
resolution concerns. Future work is slated for building a complete cognitive model of the 
submarine AO (Kirschenbaum, 1997). 

C. SUMMARY 

The issues presented are indeed complex, but manageable. Current HIP models, 
NDM paradigms, SA measurement methods, Expert vs. Novice distinctions, and their 
implication in modem DSS design have been applied to military settings successfully in a 
variety of areas (Howell, 1998). Most notably, the construction of TADMUS and most 
modem military aircraft cockpits. Only recently however, have issues related specifically 
to submarine man-machine interface been freely discussed. 

The applicability of the forgoing discussion to the AO task can be summarized as 
follows. By acknowledging and operating within the limitations inherent in the HIP 
model, it is postulated that AO's employ the NDM paradigm for decision-making in high- 
stress environments. Measures of AO SA, built in response to stored schema fostered 
during in the course of initial and periodic training, differentiate between levels of 
proficiency and skill. These levels are typically correlated to the established designation 
of expert vs. novice. After revealing these expectations of proficiency, data analysis on 
AO informational usage can be utilized in the prediction of future success and the 
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construction of tools, i.e. displays, that have the potential for improving the AO's ability 
to execute their primary task, i.e. to search out and destroy an adversary. These same 
tools can also be used as a selection and training aid in the course of an AO's professional 
development. 
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Ill METHODS 



A. RESEARCH APPROACH 

This research involves the analysis of a human performance database collected 
during the employment of a Submarine CCS Simulator. The database contains both 
numerical data regarding the use of different displays and information available in the 
simulator as well as specific background data on the test subjects. Statistical regression 
techniques determine which demographic variables can be used to predict the quality of 
an engagement. Trends in data usage among subject groupings are identified and then 
used as inputs for the design of the next generation Submarine CWS. 

B. DATA COLLECTION 

1. Subjects 

Qualified active duty Submarine Warfare officers of varying experience, as 
measured in years of Naval service and sea service participated in the study (See Table 
3.1). They represent approximately 10% of the total available pool of U.S. Navy AO's. 
Subjects trials were conducted in the Operations buildings of NSB Pearl Harbor, HI and 
NSB Bangor, WA during period of May 18 th to June 4 th 1998. Subjects were briefed on 
the purpose of Project NEMO and provided informed consent in accordance with the 
ethical conduct for subject participation specified in the Protection of Human Species, 
Secretary of the Navy (SECNAV) instruction 3900.39b (See Appendix B). 



YEARS 


MEAN 


STD DEV 


MIN 


MAX 


NAVAL SERVICE 


13.22 


5.10 


5.00 


23.00 


SEA SERVICE 


6.26 


2.92 


1.50 


14.00 



Table 3.1 Subject Experience. 



27 



2. Apparatus 

A Macintosh PowerBook 2400c/180 (603 E) equipped with an 11.1” active matrix 
screen presents the stimulus. Subject interaction with the simulation, to include 
vocalization, is recorded via a PC-to-VCR s-port adapter, a Sony HI-8 TR9 camcorder, 
and a lapel microphone. Subjects are seated in front of the computer and are allowed free 
movement during the testing. Figure 3.1 shows the equipment configuration. To avoid 
any possible distraction, the entire apparatus, with the exception of the laptop and mouse, 
is covered with an opaque black cloth. 




Figure 3.1 Experiment Equipment Configuration. 

3. Stimulus 



The simulation mimics the current tactical displays onboard a generic U.S. 
submarine. Ten separate displays are presented; OwnShip-Control (OSC), Sound 
Velocity Profile (SVP), Line of Sight (LOS), Geo-Plot (GEO), Time-Bearing (TB), 
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Time-Frequency (TF), Spherical Array Passive BroadBand (SAPBB), Towed Array 
Passive BroadBand (TAPBB), Towed Array Passive NarrowBand (TAPNB), and 
Primary Mate (MATE). The simulation contains one practice and four actual test 
scenarios. The four scenarios are representative of the expected tactical encounters for 
which U.S. AO’s train. However, the difficulty of the scenarios is purposely skewed 
toward the easier side in order to facilitate increased user interaction. The simulation 
language is Mac Common LISP. 

4. Instrument 

The instrument consists of two elements. The first is the layering of a MSExcel 
OLE compliant recorder underneath the stimulus. Informational use by the subject is 
recorded by what is referred to as a “poor man’s eyetracker.” This instrument works in 
the following manner: 1) Each display is accessed individually by use of a vertical task 
bar on the side of the “master” display. 2) All information on the displays is normally 
covered by the use of “black boxes” generated by simulation. To view a piece of 
information on a specific display, the subject places the computer-generated cursor over 
the display area to be examined and depresses a button. Once the button is depressed, the 
black box for that specific piece of information is removed and the subject is allowed to 
view the requested information. 3) The information is displayed only as long as the 
cursor is held steady. Layering the recorder under the stimulus allows for the archiving 
of the events that occur during the course of the experiment. The archived data for each 
subject trial is then stored for future examination. The second instrument is a small 
survey questionnaire designed to measure subject satisfaction as well as the perceived 
difficulty and fidelity of the stimulus. The survey also measures the subject utilization of 
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actual CCS displays and provides an avenue for subject input to the CWS design. (See 
Appendix D) 

5. Procedure 

The protocol guides the interviewer through the process of subject identification, 
indoctrination, testing, classification, and subjective evaluation. Subjects are identified 
by the interviewer via a sequenced 3-digit number and that number is used throughout the 
remainder of the testing process. Subject indoctrination consists of two phases. First, 
subjects view a short, 60 second, computer video on how to play a tic-tac-toe game in 
which the computer verbalizes it’s thoughts and actions. Next, they are allowed to play 
the same tic-tac-toe game and are instructed on how to verbalize their own thoughts and 
actions. They are then instructed to continue verbalization throughout the remainder of 
the testing process. Next, subjects are taken on a "guided tour" of the simulation by the 
interviewer and instructed on how the simulation works. A short, 15-minute, free-play 
period then ensures that the subject is "comfortable" with the actual operation of the 
tactical simulation. The testing phase commences with each subject receiving the same 
mission statement shown in Figure 3.2. 

MISSION 

"The world situation is that a war has broken out between Russia and Ukraine 
over control of the former Soviet Navy. Both nations are trying to get the US involved 
and have been attacking U.S. shipping with submarines and then blaming each other. 

You are patrolling in the eastern Caribbean Sea. Your mission is to protect U.S. shipping 
lanes. Your orders are to search and destroy any enemy submarines in the vicinity of 
U.S. shipping. The month is April and the sea state is 2." 

Figure 3.2 Mission Statement. 
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Each subject attempts to complete two of the four available scenarios. Each 
scenario starts with a status report similar to that shown in Figure 3.3. 

STATUS REPORT 

"You hold one merchant bearing 300 degrees, but no subsurface contacts. The 
broadband range of the day for a subsurface contact is 3-6 thousand yards. Intelligence 
has indicated an Alfa-type Formerly Soviet Submarine is patrolling in the area. There are 
no friendly submarines in your patrol area." 

Figure 3.3 Status Report. 

The four initial contact scenarios differ only in initial contact bearing, course and 
speed. The number of contacts and intelligence report remains the same throughout the 
experiment (See Appendix C). 

At the conclusion of the testing phase, the subjects provide background data on 
themselves and complete a subjective evaluation of the simulation. The final step is the 
subjective design phase. At this time the subject is asked to produce a sketch of what the 
subject "wants" on the next generation of Submarine CWS if they were the one in charge. 
Discussion is encouraged during this time with the tester as to facilitate creativity and 
innovation (See Appendix B for the full protocol). 

6. Experimental Design 

A 3 x 2 x 4 modified Latin square design is used in the experimental protocol (See 
Appendix C). The three refers to the 3 levels of operator proficiency (Expert, Mid, 
Novice). The two refers to the run order (First vs. Second) of the two scenarios 
presented. And the four refers to the designation of the scenario (Cl, C2, Zl, Z2) 
presented. The modified design was chosen because the distribution of the subject 
proficiency was not known in advance of the data collection. This structure allows for a 
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balanced design across the levels of subject proficiency, presentation order, and target 
scenario. 

C. DATA ANALYSIS 

1. Data Generation 

Five events are recorded by the first element of the instrument: the specific 
display viewed, the specific information viewed on the display, the total time spent 
viewing the information, any inputs the subject made to the simulation, and the time at 
which the viewing of or input to the specific display occurred. Each event is stored as a 
separate line of ASCII data and the scenario in total is stored as an individual computer 
file. The instrument's second element, the survey questionnaire, consists of four closed- 
end and three open-ended questions. The four closed questions measure scenario realism, 
scenario difficulty, display effectiveness and display utility. Three of the closed 
questions use a 5-choice Likert scale while the last uses a 10-choice ordinal ranking 
layout. The three open questions measure overall subject response to the experiment, 
subjective desires for CWS attributes, and general feedback. The questionnaires are 
completed by the subjects in the presence of the interviewer. 

2. Data Tabulation 

The computer generated ASCII files are first individually converted into an MS 
Excel spreadsheet and a Visual Basic (VB) Macro is applied (See Appendix F). The 
individual trial spreadsheets are combined together with the survey responses and then all 
of the data is merged to form a "master" spreadsheet which is organized by both subject 
and trial number to facilitate statistical analysis by demographical as well as scenario 
distinctions. The master spreadsheet is available in both MS Excel and SPLUS formats. 
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3. Variable Description 

The predictor variables are of two categories: one, demographic variables 
describing the subjects, and two, CCS display usage patterns and subject survey results. 
Subject demographic variables include six categorical variables: Position (POS; 1-Expert, 
0-Novice), Ship Type (TYPE; 1-SSN, 0-SSBN), Geographical Location (LOC; 1 -Pearl 
Harbor, O-Bangor), Duty Type (ROT; 1-Sea Duty, 0-Shore Duty), Level of Education 
(EDU; 1-Masters/Ph.D., 0-BS/BA), and Scenario Completed (SCEN; 1-C1/C2, 0-Z1/Z2). 
The display usage variables are functions of four basic variables: the number of display 
accesses, the total time of display use, the average time of display use per access, and the 
standard deviation of time of display use per access. These four variables combined with 
the survey results produce 92 distinct display usage variables. The demographic and 
display usage variables are extracted from the original ASCII files by use of the 
previously mentioned VB Macro. 

The response variables take the form of two continuous objective and one binary 
subjective MOE's. The two continuous objective measures are solution accuracy at time 
of fire (TOF) and weapon to target closest point of approach (CPA). To obtain an 
objective measure of accuracy at TOF, the absolute distance between the actual target 
location, as recorded by the simulation, and the perceived target location, recorded as the 
firing solution of the subject just prior to TOF, is computed. To standardize the geometry 
across all subjects, a 2-D grid is superimposed on the engagement area with the AO at the 
center (0,0) and the targets, actual and perceived, identified by their relative positions on 
the grid. This error, termed Composite Error (CE) is given in Equation 1. 
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CE = ((Xa-Xp) 2 +( Y a- Y p) 2 ) 1/2 [yds] 
where, 



Eqn. (1) 



(Xa, Ya) : Actual target position [yds] on the 2-D grid 
(Xp, Yp) : Perceived target position [yds] on the 2-D grid 
The calculation of weapon to target CPA is determined via a two-stage process. First, the 
final firing solution is extracted from the simulation and the appropriate weapon 
kinematic parameters are generated. The weapon is assumed to be a standard U.S. Navy 
MK-48 mod IV Heavyweight Torpedo (See Appendix E). From there, a GAMS 
simulation determines the actual torpedo to target CPA (See Appendix G) in terms of the 
CPA range (Rcpa) and CPA time (Tcpa). 

To obtain the subjective assessment of subject performance, three factors are 
examined: torpedo to target CPA, AO exposure, and time of engagement. These three 
factors answer the following questions. First, would the final solution have resulted in a 
HIT? Second, did the AO minimize own-ship’s EXPOSURE to counter-detection by the 
opposing submarine? And third, did the AO, upon achieving a “good” firing solution, 
realize that the engagement could end at that time and thus minimize the TIME of the 
engagement? It is assumed that if all three factors are within the acceptable ranges, the 
engagement as a whole would be successful, i.e. the scenario resulted in the destruction 
of the enemy and the survival of own ship. If any one is outside the prescribed range, 
then the engagement as a whole would be considered unsuccessful, i.e., the scenario 
resulted in the survival of the enemy and/or the destruction of own ship. 

The notion of the AO getting a weapon hit is the first factor examined. A HIT is 
defined to have occurred if the final solution generated weapon parameters would have 
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resulted in an impact with the enemy. This factor, classified as a 1 for an impact and a 0 
otherwise, is determined via a two-stage process. First, the final firing solution entered 
by the subject is extracted from the simulation. And second, as with the determination of 
CE, the appropriate CPA parameters are generated. If the CPA range is less than acoustic 
acquisition range for the torpedo (2000 yds) and the CPA time is within the weapons run 
time (3500 sec), the weapon is assumed to impact the target. 

AO exposure is the next factor examined. EXPOSURE, in this case, is defined as 
the minimum range experienced by the AO during the course of the engagement. Two 
factors are used in determining a cutoff threshold. First, it is assumed that the enemy is 
pursuing the AO and while it is assumed that the AO has an acoustical advantage, the 
chance of counter-detection increases as exposure decreases. Second, to coin a movie 
phrase, "the problem with submarines is that they do not turn on a dime" (Clancy, 1990). 
The point being that, as two submarines get close to one another, the risk of collision is 
just as important as the risk of counter-detection. For these two reasons, a lower limit of 
2000 yards is placed on EXPOSURE. If the AO to enemy range is greater than 2000 
yards for the duration of the problem, a 1 is recorded and if not, a 0. 

The time required to get a solution is the final factor examined in determining 
overall solution quality. While longer engagements can lead to more accurate solutions, 
they also increase the danger to the AO. Remember that while the AO is pursuing the 
enemy, the enemy is also pursuing the AO. Hence the danger of counter-detection by the 
enemy, and resulting counter-fire, increases as the AO spends more time in the vicinity of 
the enemy. A perfect solution obtained after the enemy has disengaged or while getting 
shot is of no value. Therefore, to account for this, a TIME constraint of 1500 seconds is 
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used to identify the point at which counter-detection and eventual counter-fire might 
occur by the target. If the AO initiated a firing prior to this time, a 1 is recorded and if 
not, a 0. 

These three factors: HIT, EXPOSURE, and TIME are combined to determine if 
the engagement is a success. If all three factors are a 1, the engagement is a success and 
a 1 is recorded for the variable SF (success/failure). If any of the factors is a 0, then the 
engagement as a whole is a failure and a 0 is recorded for SF. 

4. Statistical Analysis 

To analyze and compare the performance of the subjects, a two pronged approach 
is used. First, performance is studied as a function of the subject demographic variables. 
For this, the two MOE’s, CE and SF, are used. Classical logistical regression (Hamilton, 
1992) and CART analysis (SPLUS, 1996) compromise the bulk of the data analysis 
techniques employed. The second prong will explore differences in display usage and 
solution generation strategies of the subjects as a function of the demographic variables. 
This approach is shown in Figure 3.4. 




The display design process will utilize a three pronged approach. First, the 
subject survey evaluation of the effectiveness and usefulness of the current CCS displays 
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(See Appendix D) as measured by the mean response will be extracted. Second, link 
analysis techniques will extract the display usage and solution generation patterns of the 
"best" group of AO's as distinguished by distributional and categorical relationships 
previously identified. And third, the top ten salient issues generated by subject feedback 
when asked of their desires for the next generation of CWS (See Appendix D) will be 
used as the final input. These three inputs will then be merged into a comprehensive 
layout and set of display attributes for the final CWS. This design methodology is shown 
in Figure 3.5. 
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IV. RESULTS 



The analysis conducted in this research is primarily exploratory in nature. Over 
100 hours of experimental data is utilized in the creation of a subject performance 
database. From the database, the variables most indicative of subject performance are 
extracted and used as the basis for analysis. Any subject modeling or statistical analysis 
results obtained will need to be verified by an independent study prior to drawing any 
final conclusions. 

A. TRIAL RESULTS 

The subject pool for the experiment consists of 49 volunteers. Of these, 36 
subjects correctly finished the testing protocol resulting in 65 completed trials. Even with 
the exclusion of 26 percent of the experiment participants, the use of the modified Latin 
square design for the remaining 65 trials succeeded in partitioning the subject pool fairly 
evenly according to the 6 demographic variables. This is shown in Table 4.1. 



Demographic Variable 


Variable = 1 


Variable = 0 


n (%) 


n (%) 


POS (1 - Expert, 0 -Novice) 


27 (42) 


38 (58) 


TYPE (1 - SSN, 0 -SSBN) 


33 (51) 


32 (49) 


LOC (1- Pearl Harbor, 0 - Bangor) 


28 (43) 


37 (57) 


ROT (1 - Sea Duty, 0 — Shore Duty) 


50 (77) 


15 (23) 


EDU (1 - MS/MA/PhD, 0 - BS/BA) 


34 (52) 


31 (48) 


SCEN (1 - C1/C2, 0 - Z1/Z2) 


33 (51) 


32 (49) 



Table 4.1 Subject Pool Partitioning by Demographic Variables. 

Subject responses to the issues of simulation realism and scenario difficulty are 
first examined to verify the original mandates of simulation fidelity and ease of use. As 
is shown in Figure 4. 1 and 4.2, the frequency histograms of Realism and Difficulty are 
right-skewed, with a median of four and a mean of 3. With the scales used, i.e. 1-Very 
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Difficult/Unrealistic and 5-Very Easy/Realistic, the results indicates that, as a group, the 
subjects rated the simulation as realistic and fairly easy to use. 




Figure 4. 1 Histogram of Scenario Difficulty (1-Very Difficult, 5-Very Easy). 




Figure 4.2 Histogram of Scenario Realism (1-Very Unrealistic, 5-Very Realistic). 

B. MOE RESULTS 

When analyzing and distinguishing the performance of the subjects, the earlier 
defined two pronged approach is used. The distributions of the MOE's, CE and SF, along 
with CPA, EXPOSURE, and TIME, are examined first in an attempt to understand the 
underlying relationships between the MOE’s. Excluding TIME, the distributions indicate 



40 



that the subjects performed within the ranges of values expected. The distribution of 
TIME, however, did not reveal the expected bias toward shorter engagement times. 

The distribution of CE is shown in Figure 4.3. As can be seen, the frequency 
distribution of CE is heavily left-skewed with most subjects having a low CE with a 
median of 1291 yds and a mean of 1386 yds. This indicates that most subjects arrived at 
a reasonably accurate solution at TOF. 




CE (yds) 

Figure 4.3 Frequency Histogram of CE. 

The distribution of the weapon to target CPA parameters, Tcpa and Rcpa, are 
shown in Figure 4.4 and 4.5. As can be seen, the frequency distributions are left-skewed 
(median(TcpA) = 127 secs, mean(TcpA) = 146 secs; median(Rcp A ) = 248 yds, mean (Rcpa) 
= 391 yds) indicating most subjects made limited use of the torpedo’s capabilities at 
TOF. Rather it seems that the subjects preferred to rely on a more accurate solution (i.e. 
a low CE) at TOF. 
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Rcpa (y<*s) 

Figure 4.5 Frequency Histogram of Rcpa- 
The distribution of exposure is shown in Figure 4.6. As can be seen, the 
frequency distribution of exposure is lightly right-skewed with a median of 4000 yds and 
a mean of 3903 yds. This indicates that most subjects had a high exposure. This is 
consistent with the expected results in that most subjects should have made a concerted 
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effort not to get too close to their adversary and therefore risk counter-detection and/or 
collision. 




Exposure (yds) 

Figure 4.6 Frequency Histogram of Exposure. 



The distribution of the Time is shown in Figure 4.7. As can be seen, the 
frequency distribution of time is fairly evenly distributed, with approximately equal 
median and mean, indicating a full spread of engagement times among subjects and no 
significant biasing toward either a long or short engagement duration. A Kolmorgorov- 
Smirnov Goodness-of-fit (KS) test for uniformity fails to reject a uniform distribution 
with a p-value of 0.76. This result is surprising considering the instruction set given to 
the participants. It was thought that the subjects as a whole would have shot much more 
quickly. Coupled with the earlier lack of reliance on the torpedo’s capabilities, as shown 
in the distribution of the weapon to target CPA parameters, the subjects seem to show a 
bias toward a more conservative approach methodology. 
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Time (sec) 

Figure 4.7 Frequency Histogram of Time. 

An examination of the probability cross-tabulations between the binary variables 
of TIME, EXPOSURE, and HIT, as shown in Table 4.2, reveals noticeable dependence. 
A x 2 test for independence indicates that the three variables are indeed dependant 
(X 2 4=10.63, p-value=.03). Therefore, subjects that succeed in one measure are more 
likely to succeed in another and visa versa. 



HIT=0 


HIT=1 


TIME\EXPOSURE 


0 


1 


TEVIEVEXPOSURE 


0 


1 


0 


0.00 


0.00 


0 


0.05 


0.05 


1 


0.02 


0.02 


1 


0.06 


0.80 



Table 4.2 HIT vs. EXPOSURE vs. TIME Probability Cross-Tabulations. 

As outlined earlier, the three binary factors of HIT, EXPOSURE, and TIME, are 
multiplied to determine if the engagement is a success. Recall that if all three factors are 
classified with a 1, the engagement is a success and a 1 is recorded for the variable SF. If 
any are classified with a 0, then the engagement as a whole is a failure and a 0 is recorded 



44 



for SF. Using that criteria, the subject population, as a group, achieved a 77% success 
rate (n = 50) and a 23% failure rate (n = 15). 

C. SUBJECT MODELS 

The next step is to determine which models, if any, can be used to adequately 
describe subject performance. Due to the availability of both continuous and binary 
response variables (CE and SF), that capture two different aspects of subject 
performance, and the six demographical predictor variables, two different models are 
developed. 

1. CE Models 

First, plots of CE and Time vs. all seven (POS, TYPE, LOC, EDU, ROT, SCEN, 
and SF) demographic variables are examined for differences in the mean value of CE and 
Time. These are shown in Figures 4.8 and 4.9. As can be seen, the differences in mean 
Time are considerably less than the differences in mean CE. This indicates that except 
for the variable SF, there appears to be no significant interaction between Time and the 
six remaining subject demographic variables. Alternatively, there are noticeable 
interactions between CE and the demographic variables of SF, LOC, ROT, and TYPE. 
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Figure 4.8 Mean CE (yds) by Demographic Variables. 
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Figure 4.9 Mean Time (sec) by Demographic Variables. 



A Welsh's modified two sample T-test is run on the differences in mean CE and 



mean Time with respect to each of the seven demographic variables to determine if the 
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results were indeed significant 1 . As was the postulated earlier, the factors of SF, TYPE, 
LOC, EDU and ROT show significant effects for CE, while only SF shows significant 
effects for Time. We note that with the number of two sample tests performed, the p- 
value becomes a descriptive statistic. Nevertheless, the p-values still indicate that there is 
"strong" evidence that there is a difference in the expected MOE's between the two 
groups. However, the exact level of significance of the test is somewhat more difficult to 
quantify. The outcome in the form of the mean values and p-values is shown in Table 
4.3. 



Response 


Variable 


Variable=0 


Variable=l 


p- Value 


Mean Time 


SF 


1557.91 


947.46 


<0.01 


POS 


1066.03 


1046.01 


0.82 


TYPE 


1084.55 


1032.35 


0.55 


LOC 


1073.93 


1038.69 


0.69 


EDU 


1023.36 


1093.53 


0.42 


SCEN 


1061.39 


1055.12 


0.94 


ROT 


1045.32 


1063.00 


0.86 


Mean CE 


SF 


1617.03 


1334.33 


0.03 


POS 


1341.08 


1458.96 


0.62 


TYPE 


1508.00 


1263.46 


0.02 


LOC 


1529.99 


1201.67 


0.01 


EDU 


1280.45 


1491.01 


0.06 


SCEN 


1371.44 


1401.86 


0.85 


ROT 


1597.68 


1312.19 


0.02 



Table 4.3 Descriptive P-values for Differences in Mean CE and Mean Time. 

Next, speed (Time) vs. accuracy (CE) is fit as a function of the seven 
demographic variables. This is accomplished by linearly regressing CE on Time with 
each of the seven demographic variables considered separately. In each case, the 
interaction of the demographic variables with the factor Time is also included. Figures 
4.10 through 4.16 show the results of these multiple regressions. As can be seen, the 
variables of SF (F2,63=4.03, p-value<01), LOC (F2,63=3.09, p-value<01), TYPE 



1 For the purposes of this research, significance is defined as p-value<=.10. 



47 



(F 2 , 63 =3.06, p-value<.01), EDU (F 2j 63=2.76, p-value=.01) and ROT (F 2> 63=2.42, p- 
value=.01) all show significant effects. The variables of SCEN (F 2j 63 =. 37, p-value=.34) 
and POS (F 2> 63=.35, p-value=.35) were negligible. These first results, particularly those 
from the POS regression, contradicts the expected notion of the distinction between 
experts and novices, and raises the issue of a possible reevaluation in the use of military 
rank as the primary designator of expert and novice. In addition, the significance of the 
ROT factor indicates that AO skills are fragile in that those subjects that utilize their 
skills on a frequent basis maintain a better mastery of the subtleties of the task, while 
those that are otherwise engaged experience a degradation in their abilities. 




Figure 4.10 Speed (Time) vs. Accuracy (CE) Regression Conditioned on SF. 
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Figure 4.11 Speed (Time) vs. Accuracy (CE) Regression Conditioned on EDU. 




Figure 4.12 Speed (Time) vs. Accuracy (CE) Regression Conditioned on LOC. 
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Figure 4.13 Speed (Time) vs. Accuracy (CE) Regression Conditioned on POS. 




Figure 4. 14 Speed (Time) vs. Accuracy (CE) Regression Conditioned on ROT. 
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Figure 4.15 Speed (Time) vs. Accuracy (CE) Regression Conditioned on SCEN. 




Figure 4. 16 Speed (Time) vs. Accuracy (CE) Regression Conditioned on TYPE. 
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Additionally, when the sign of the slope of the regression line was examined in 
under all possible combinations of subjects (2 7 ), only one combination (POS=0, LOC=l, 
TYPE=1, EDU=1, ROT=l; n=6) showed a negative value. A binomial sign test applied 
to the null hypothesis that the probability of a negative slope is 1/2 against the alternative 
hypothesis that the probability is not 1/2, finds significance in that in no case does a 
longer engagement lead to a more accurate solution. Therefore, it appears that an AO's 
best solution is the first, and any further refinement may be of little value. 

Finally, a full model fitting CE vs. all seven demographical variables and their 
interaction with Time is examined. Using Mallow’s Cp, a stepwise linear regression is 
conducted. The variables of SF, Time, and SF x Time comprise the best-fit model with a 
multiple correlation coefficient of 0.31. Table 4.4 shows the estimated coefficients of the 
predictor variables and the associated p-values remaining in the “best fit” model. These 
results loosely correspond with the earlier results obtained from the examination of mean 
CE and the speed vs. accuracy trials. Examination of the full model reveals that the 
highest correlation possible between CE and the fitted CE occurs with value of 0.45. 
Therefore, with the three factors in the best-fit model, 69 percent of the total possible 
model correlation is attained. 



Variable 


Coefficient 


SE 


p-Value 


SF 


1036.86 


451.11 


0.02 


Time 


1.06 


0.17 


<0.01 


SF x Time 


-0.75 


0.49 


0.10 



Table 4.4 “Best Fit” Linear Regression Model Coefficients and p-Values. 
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2. SF Models 



The second model examined is a logistic model with factor SF as the binary 
response and the six subject demographic variables (EDU, LOC, POS, ROT, SCEN, and 
TYPE) as the binary predictors. Construction of the model follows two separate 
methodologies. 

First, the model is fit with the full data set. This allows for the estimation of the 
expected probability of success for each subject based on the particular combination of 
predictor variables. Since these probabilities range from 0 to 1 and the actual response 
variable was binary, all fitted probabilities greater than 0.50 are assigned a value of 1 and 
all those probabilities less than 0.50 are assigned a value of 0. The model is tested by 
examining the correlation coefficient between the actual and predicted success vectors. 
As shown in Table 4.5, the factors of TYPE, LOC, and POS have the greatest impact 
with a model correlation of 0.81. 

Second, the data was randomly split into two separate sets each of equal size. The 
model was fit to half of the data set and then applied to the other half in the prediction of 
success. This was repeated 10 times for standardization. As shown in Table 4.5, the 
average correlation between actual and predicted success for the 10 repetitions is .76. As 
with the full model, the factors of TYPE, LOC and POS have the greatest influence. In 
both the full and ten random models, errors in the prediction of success are predominately 
due to an overestimation of a subjects abilities rather than underestimation. 
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VARIABLE 


BETA 


FULL MODEL 


10 RANDOM MODELS 


POS 


PI 


0.87 


1.32 


TYPE 


p2 


5.71 


5.13 


LOC 


P3 


-5.57 


-4.91 


EDU 


P4 


-0.04 


0.06 


SCEN 


P5 


0.72 


0.81 


ROT 


P6 


0.68 


0.71 


Correlation coefficient 


r 


0.81 


0.76 



Table 4.5 Regression and Correlation Coefficients for the Full and Random SF Models. 

In an attempt to visualize the results of the logistic regression model of subject 
performance, CART techniques (SPLUS, 1997) are utilized. Figure 4.17 shows the 
classification tree for the subject performance model. The estimated probability of 
success (Ps) for each leaf is at the terminal node. Due to the nature of the data, i.e. binary 
values independent of each other, the CART tree is ideal for quickly identifying a subject 
by their demographic variables and their associated probability of success. For example, 
for an AO from Bangor (LOC=0), with a BS degree (EDU=0), attempting scenarios 
C1/C2 (SCEN=0), we follow the tree down and to the left to get an estimated probability 
of success of 0.75. As with earlier models, the factors of LOC, EDU, and ROT are 
significant while yet again, the factor of POS is negligible. The misclassification rate for 
the CART model was .22. When compared to the correlation coefficients arrived at with 
the logistical model (r=0.81, r=0.76), we find that the CART analysis arrives at similar 
conclusions. 
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Figure 4.17 CART Estimated Probability of AO Success based on Demographic 

Variables. 

The second approach examines the relationships between the 92 dependent 
variables of display usage and solution generation, and the 7 (SF, POS, LOC, EDU, ROT, 
TYPE, and SCEN) independent demographic variables. Because the measurement scales 
of the dependent variables differ considerably, each dependent variable is standardized 
prior to analysis. The distributional differences of the 92 dependent variables are 
examined in two ways. First, to see if the expected value of the dependent variables 
differ with the level of the independent binary demographic variables, two sample T-tests 
are run on the 92 dependent variables as a function of the independent variables. The 
outcome of this is a 7 x 92 matrix of p-values indicating differences in the mean value of 
each of the 92 variables as a function of the seven demographic variables. Box-plots of 
the 92 p-values for each of the demographic variables are shown in Figure 4.18. 
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Figure 4.18 Boxplots of the 92 Display Usage and Solution Generation p-Values. 

However, in testing 644 (7x92) hypothesis, both random statistical noise and 
actual statistical differences are expected to be present. If there is no difference in say 
LOC, then we would expect the 92 p-values for LOC to be uniformly distributed between 
zero and one. By examining Figure 4.18 it appears that the p-values for all demographic 
variables, with the exception of POS and SCEN, are less than expected. To confirm this, 
for each of the independent variables, a Kolmorgorov-Smirnov Goodness-of-Fit test for 
uniformity is applied to the 92 p-values to test to see if any significance that was found 
was due to actual differences in the independent variables, and not due to random 
statistical noise. Secondly, to verify these results, a x 2 test was run on the actual 
distributional relationships among the independent variables. The results are shown in 
Table 4.6. As can be seen the variables of SF, LOC, TYPE, and SCEN were significant 
at the a=05 level and the variables of EDU and ROT was significant at the a=.l level. 
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Surprisingly, the variable of POS was not a distinguishing trait in either of the two tests. 
This gives added weight to the earlier conclusions of the linear regressions. 



VARIABLE 


p-Value 


KS 


x 2 


SF 


0.01 


0.04 


EDU 


0.06 


0.09 


LOC 


0.01 


0.01 


POS 


0.74 


0.53 


ROT 


.05 


0.08 


SCEN 


0.01 


0.01 


TYPE 


<0.01 


0.01 



Table 4.6 KS and % 2 p-Values for Display Usage and Solution Generation Differences. 

D. DISPLAY DESIGN 

1. Data Input 

The basic foundation of human interface display design is to model the display 
after the expert, or "expected", operators (Bailey, 1980; Wickens, 1992). In this 
experiment, however, the designation of “expert” operator has shown a lack of significant 
performance distinction. Therefore, the “successful” operator is used instead. This way, 
patterns of behavior that cut across the demographic differences of AO’s are exploited for 
use in developing the underlying principles for CWS design parameters. 

A single step link analysis is performed upon the patterns of display usage and 
solution generation of the successful operators. The patterns of display usage are 
categorized as belonging to a 10 x 10 matrix. The number of transitions from each one of 
the 10 displays to another is counted and then standardized into subject specific 
frequencies with the constraint that all of the 100 frequencies add to one. The 10 x 10 
matrix is then collapsed to a 4 x 4 matrix according to the coding scheme as follows: 
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SONAR (SAPBB, TAPBB, TAPNB), DERIVED (TB, TF, GEO, LOS), ENVIRON 
(OSC, SVP), and SOLN (TMA). The result of the link analysis is shown in Figure 4.19. 
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Figure 4.19 Successful AO Display Usage Link Analysis. 



As can be seen the successful operator utilizes a unique distribution of display usage in 
the course of arriving at a solution. The three links of SONAR-SONAR, SOLN- 
DERTVED, and DERTVED-SOLN account for more than 50% of the total links. 

A single step link analysis is also performed upon the patterns of solution 
generation of the successful operators. The patterns of solution generation are 
categorized as belonging to a 4 x 4 matrix with the elements of target Bearing, Range, 
Course, and Speed, as the anchor points. The number of transitions from each one of the 
4 elements to another were counted and then standardized into subject specific 
frequencies with the constraint that all of the 16 frequencies add to one. The result of the 
link analysis is shown in Figure 4.20. As can be seen, there are no obvious biases or 
patterns in the progression of links. Hence, the absence of a distribution is, in and of 
itself, important. 
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Figure 4.20 Successful AO Solution Generation Link Analysis. 

Close examination of the procession of both link analysis frequency distributions 
show that in both cases, successful subjects follow the standard contact-reporting liturgy. 
That is, from Bearing to Range to Course to Speed. Deviation from this progression 
appears to be detrimental to the outcome of the engagement. This deviation could be 
responsible for actually inhibiting the correct decision-making and schema forming 
process. 

2. Survey Input 

Subject ratings of current CCS display effectiveness and usefulness are measured 
with both a five and a ten-point ordinal ranking scheme. The mean rating of each display 
is compared against the others in the determination of the relative position of each 
display. Again, the displays are recoded using the same methodology as in the link 
analysis. The results are shown in Figure 4.21 and 4.22. In the measurement of utility, 
the ENVIRON displays were at the top while the SOLN displays were at the bottom. In 
the measurement of effectiveness, the SONAR displays were at the top while the 
DERIVED displays were at the bottom. These results indicate that certain CCS displays 
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are actually inhibiting the users SA by failing to provide the required information in the 
proper format. This also could inhibit the proper development of schema's. 





Figure 4.22 Subject Response for the Mean Usefulness of CCS Displays. 

Subject response to the open-ended question regarding desires for the next 
generation of CWS was mixed. Approximately 30% of the subjects did not complete the 
question in any form. Of those that did, only 10% provided a written response. Most 
subjects, 60%, preferred to interact with the interviewer in a “brainstorming” session. 
The sessions typically lasted 20 minutes with one lasting as long as 2 hours. Their 
responses were manually recorded and then analyzed for salient features. A salient 
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feature is defined as a point, or issue, of focused interest by the subject for an attribute, or 
ability, that is desired of the CWS. Additionally, an in-situ drawing was kept by the 
interviewer and presented to the subject at the end of their input in an attempt to elicit 
further input and/or criticism. This drawing was updated after each subject’s input and 
evolved in response to the subject pool desires for the CWS. A frequency analysis of the 
subject generated salient feature is shown in Table 4.7. As can be seen the top five 
attributes were the availability of history, multiple tasking capabilities, information 
standardization, alternative evaluation, and single point origination of information. 



FEATURE 


FREQUENCY 


Availability of history 


83 


Multiple tasking capabilities 


75 


Information standardization 


64 


Alternative evaluation 


63 


Single point origination of information 


59 


User defined modes 


54 


User defined layout 


50 


Increased use of computer algorithms 


50 


Increased use of cots 


45 


Electronic publications 


40 



Table 4.7 Frequency of Salient Features Generated During Subject Feedback. 

The top five features desired by the subjects indicate a desire for the CWS to be 
fully integrated with the current support systems internal to the submarine as well as 
providing for relief from the current memory intensive processes now required of AO’s. 
In addition, most subjects made the comment that it would be advantageous and worth 
the expenditure required to have displays available for use that were based on technology 
less than 10 years old. 
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V. DISCUSSION AND CONCLUSIONS 



This thesis had three primary objectives. The first objective was to explore 
possible relationships between both subject demographic differences and CCS 
employment strategies in the prediction of a successful engagement. The second 
objective was to provide data in the development of a notional "AO friendly" CWS 
interface. It is postulated that this interface will allow the AO to more accurately 
evaluate relevant tactical information with less effort through the use of more appropriate 
information presentation and display formatting. Finally, the third objective was to 
provide for principles which could be applied in the areas of personnel selection, 
qualification, and training. 

The areas of HEP, NDM, situational awareness, schema theory, expert vs. novice 
performance and the principles of display design were examined in detail. In particular, 
the military applications, past and present, of the six theories were discussed. As noted, 
most of the previous work has been used in pilot/aircraft interface analysis, and was not 
specific to submarine AO’s. However, as in Hooper (1994), we postulate that the gap 
between the two is so slight as to infer parallel results. 

A computer-based simulation was created and presented to approximately 10% of 
the active duty submarine AO’s. The subjects were run though two of four pre-scripted 
initial contact scenarios in a modified Latin square design. Two objective and one 
subjective evaluations of performance were obtained. Solution accuracy at TOF and 
eventual weapon to target CPA were continuous objective measures, while the 
determination of a “successful” engagement was subjective. In addition, the subjects 
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participated in a design survey to facilitate feedback on not only the simulation presented 
to them, but also specific desires for the next generation of CWS. 

An exploratory statistical analysis was performed on the subject-generated 
database with the goal of determining if subject demographic differences or CCS usage 
strategies are predictive of subject performance. Both logistic and linear regression 
techniques were applied to the measures of SA obtained for the subjects. In the 
prediction of a successful engagement, the factors of geographical location, ship type, 
and duty type were the three most important determinants. While these three attributes 
are not surprising when taken as a group, their order of relative importance and the 
absence of the distinction of Expert or Novice was not expected. This coupled, with an 
almost zero correlation between CE and number of years at sea, a synonym for expert 
designation, raises the question as to whether a good AO is “born or made”. Analysis of 
linear relationships between time of engagement vs. CE (Speed vs. Accuracy) reveals two 
important observations. First, the shorter the engagement, the more accurate the solution 
that is shot. And second, once again, expert vs. novice distinction is negligible, while 
geographical location, ship type, education, and duty type are clearly observable effects 
in terms of the MOE's utilized. It was then shown via application of Welsh’s modified 
two-sample T-test coupled with Goodness-of-fit tests that factors of SF, EDU, LOC, 
ROT, and TYPE do in fact demonstrate variance in the cognitive progression of links in 
both display usage and solution generation. This lack of a distinction between expert and 
novice operator conflicts with the results expected from the literature. Perhaps, the core 
distinction between experts and novices cannot be gleaned from something as simple as 
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rank. The effects of the demographic variables conclude that the distribution of AO skill 
is not evenly spread out across such factors as geographic location and ship type. 

A single-step link analysis was examined on both display usage and solution 
generation. The question of selecting which subject distinctions to condition on was 
solved by only examining the differences between successful and unsuccessful subjects. 
The successful subjects were shown to follow standard submarine liturgy and submerged 
approach methodology (as explained in Chapter 4), while unsuccessful subjects were 
observed to be going about their approach in a seemingly ad hoc manner. This ad hoc 
manner, commonly referred to as “Easter-egging," exhibited no definable sequence or 
pattern of informational usage. This further reinforces the benefits of reliance on 
established methodologies and techniques. Rather than attempt to develop their own 
individual style, AO's should instead concentrate on techniques that have been proven to 
work. As the saying goes, "all you need is rhythm." Finally, results from the two link 
analyses were fused with both subject evaluations of the current CCS’s, and subject 
desire for attributes of the next generation of CCS in the construction of a preliminary 
CWS design. While the original mandate envisioned that subject input would only be 
made for the fire control module portion of the CWS, subject dissatisfaction with the 
current CCS’s and the strong desire for a revolutionary leap in CCS employment, 
significantly expanded the scope of subject input. The final product therefore, is touted 
to be an attempt to describe the initial design parameters for a complete CWS 
encompassing all aspects of AO employment. 
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A. 



NOTIONAL CWS DESIGN 



To reiterate, a 3-pronged display design approach was utilized in the development 
of the principles for the CWS. This approach combined the subject survey evaluations of 
the effectiveness and usefulness of the current CCS displays, the link analysis extraction 
of the display usage and solution generation patterns of the "best" group of AO's as 
distinguished by the distributional and categorical relationships identified, and the top ten 
salient issues generated by subject. The principles and parameters developed are 
presented as a basis upon which future research can capitalize. 

The physical layout of the CWS is founded on two, side by side, large (i.e. 19-23 
inches) computer touch screens. Touch screens were selected for their ease of use and 
quick response capabilities. The left CWS display, shown in Figure 5.1, consists of four 
fixed modules with an optional fifth for submarines with strategic commitments. The 
GEO-PLOT and CRO modules share the same function as the modules of the same name 
in TADMUS. The O/S CONDITION module displays all actual and ordered kinematic 
parameters (course, speed, and depth) for own-ship. The SVP module is a graphical 
representation of the current ocean environment. Ship’s depth would be displayed 
graphically within the appropriate current SVP. The optional module for the left CWS is 
an event timer for submarines with strategic commitments. This option was not desired 
by the non-strategic community but was deemed important by the strategic community. 
This timer would operate in the same manner as the response manager in TADMUS. The 
location and size of the modules was determined by examination of the data and survey 
results. The modules are arranged to facilitate increased situational awareness by 
allowing for a more natural information flow between the system and the operator. This 



66 



then allows for a reduction in the cognitive workload of secondary tasks that are currently 
inhibiting the AO from forming the necessary schema's. The now liberated cognitive 
resources can be applied back to the AO's primary task, i.e., approach and attack of the 
enemy. 





Figure 5.1 Left Side of Notional CWS Display. 

While the left side of the CWS is rigid in structure, the right side is user 
determined. This flexible display concept is primarily an outgrowth of desire among 
AO's that the CWS be adaptable to a variety of situations and scenarios. This side of the 
CWS is split into three modules: an upper, a lower, and a series of selection buttons. The 
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right CWS is display is shown in Figure 5.2. The selection buttons determine which of 
the four upper sub-modules (SONAR, DAMAGE CONTROL, ENVIRONMENT, and 
ERP) and four lower sub-modules (FCS, RADIO, WEAPONS, and ENGINEERING) are 
displayed. The selection buttons operate such that only one of the upper sub-modules 
and one of the lower sub-modules can be displayed at any one time. The sub-modules 
are organized such that in most cases, no two upper or lower sub-modules will have to be 
displayed at the same time. 

II ■ | 

| 
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UPPER DISPLAY 



LOWER DISPLAY 




Figure 5.2 Right Side of Notional CWS Display. 
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The details of the sub-modules, with the exception of the fire control system sub- 
module (FCS), are not presented in this study due to space and time limitations. 
However, they were presented to the sponsor by separate correspondence. The FCS sub- 
module, shown in Figure 5.3, has four distinct areas each with a different purpose. 




Figure 5.3 FCS Sub-Module of Notional CWS Display. 



Area one is a standard contact report listing of all current information on the 
target of interest. The order is top to bottom, and is presented in numeric and bullet 
format to aid in rapid and detailed recall. Area 2, shown in Figure 5.4, itself consists of 4 
areas. The solution area gives the current system solution to the target selected. The 
BRDD (Bearing Rate Difference Dots) and RRDD (Range Rate Difference Dots) areas 
are the same as the current iteration with the exception that they are standardized in their 
orientation. Currently, the time reference alternates between top-down and bottom-up. 
In the CWS, both would be top-down time referenced displays. The LOS area gives the 
standard 3 -element representation of the current target geometry. 
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Figure 5.4 FCS Sub-Module - Area II. 

Area three is the same as two, with the exception that area two is keyed to the 
current system solution and area three is keyed to the current trial solution. This structure 
allows for the viewing of both data representations in the same foveal area and therefore 
a reduction of the memory workload of the AO. This also speeds up the AO's decision- 
making process by reducing the waiting time required to make the mental comparisons 
between the current trial and system solutions. Area four is a window for viewing the 
FUSION plot. This plot, currently undergoing experimental evaluation, is a merging of 
the current TB, TF, GEO, and ENVIRONMENTAL plots. This plot allows for a 
“fusion” of all current tactical and organic information pertinent to the target selected. 
One additional attribute of this plot that was requested above all others was that it have a 
“time scaling” feature. In other words, the AO would have the ability to look back in 
history to allow for an EBR approach to the target’s intentions. Such a display might 
look similar to Figure 5.5. 
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Figure 5.5 FCS Module - FUSION. 

While the CWS’s primary purpose is to aid in the completion of the AO’s task, 
the expanded role desired by the subjects raises issues concerning its physical placement. 
The original CWS was envisioned to only be available in the submarine control room, 
however, with the added capabilities, it should be placed at several strategic locations. In 
particular, repeaters in the CO/XO stateroom and the officer wardroom would add in the 
process of maintaining situational awareness of the tactical and strategic situation by the 
submarine crew. 

This notional CWS design should result in the following: First, significantly 
reduced processing time by presentation of tactical information in a natural and consistent 
manner. Second, an increased CCS functionality by allowing for a broader range of 
information to be available to the AO. Third, an increased rate of user interaction by 
allowing the user to specify the type and time of information presentation. And fourth. 
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an increased user acceptance of the system due to the fact that it was the users that 
determined CWS's form and function. 

B. LESSONS LEARNED 

Three lessons learned during the course of this research are worthy of mention: 

1. The use of a “poor man’s eye-tracker”, while solving many of the difficulties in 
the data collection phase, was a point of serious consternation. Subject 
dissatisfaction was very pronounced and several subjects were outwardly hostile. 
The use of a head-mounted eye-tracker, while expensive, would be a better 
solution. 

2. The simulation employed was limited in scope and fidelity by certain software 
design issues and constraints. Full employment of available hardware capabilities 
to ensure maximum realism and usefulness must be a priority before further fleet 
evaluation. 

3. Analysis of the data generated indicates several areas meriting serious inquiry. 
Chief among them is the reevaluation of Expert/Novice designation. Issues 
concerning relevant display design issues need to be examined early in the 
procurement process in order to achieve maximum benefit. 

C. RECOMMENDATIONS FOR FURTHER RESEARCH 

Project NEMO shows great promise for Submarine commanders at sea operating in 
the high-stress, contact dense and politically ambiguous environments that the future 
surely promises. However, five further research issues merit consideration: 

1. Use of the database for further analysis. Only a few data analysis avenues and 
hypotheses were examined in this thesis. Future work by NUWC, working in 
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conjunction with staff and students at NPS, should continue to explore the wealth 
of data collected during the course of this experiment. In addition, several areas 
were not explored due to the limitations of subject pool size, composition, and 
statistical significance. Continued data collection, perhaps from East Coast based 
submarine AO’s, would provide the necessary database to alleviate this. 

2. Continued CWS development. NUWC and ONR should seriously consider the 
principles developed for the preliminary CWS design proposed for further 
evaluation and possible implementation in tools such as Micro-SAINT (Wickens, 
1992), with the eventual goal of formal presentation to the NSSN program office. 
This preliminary design, constructed for the first time by actual operators, should 
result in improved AO situational awareness, increased tactical proficiency, and 
greater overall operator satisfaction. 

3. Cognitive modeling of novice system users . As stated earlier, the time constraints 
imposed by rapidly unfolding scenarios will surely be measured in minutes, 
possibly seconds, rather than hours or days. The systems must be able to be used 
effectively by the all levels of operators, since it is the Junior Officer who most 
likely will have the watch when hostilities commence. This would entail a 
reevaluation of the system from the "novice" point of view in order to facilitate 
maximum throughput of key tactical information. A mode command or toggle 
device installed into the system could be used to shift between levels of user skill. 

4. Use of NEMO for Command Level training. The nature of command in a 
submarine control room is that the users operate in a windowless environment 
devoid of external stimuli or reference to the outside world. This environment 
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can be replicated quite easily via simulators. The simulators, based at major shore 
installations as separate entities or onboard the units themselves in the form of 
“canned training scenarios”, would provide for the proper training of current and 
future commanders. The simulators could also provide some sort of diagnostic 
decision making performance measures that could be compiled automatically to 
indicate the skill of the user. 

5. Use of NEMO for Command selection. Selection for command at sea, while 
monumental for the selectee, is at best a vague and uncertain process for selection 
boards. Frequently board members have little or no personnel experience or 
knowledge of the selectee and so they base their decision on paper reports, 
second-hand recommendations, and “sea stories." The CWS developed by 
Project NEMO could be used to supplement the board's decision by testing the 
potential commander's innate decision making process. That process could then 
be categorized to see if the selectee possesses the correct cognitive processes 
required of “successful” commanders. The selectee would be run through a 
scenario specifically crafted by the board and his, or her, outcome would aid in 
the determination of adequacy for command. Alternatively, since cognitive 
abilities are for the most part "bom" and not made, the systems could be used for 
program entrance selection across the entire range of warfighter roles. 

In conclusion, this thesis has provided insight into the cognitive processes of the 
AO. The proposed CWS, in addition to assisting the AO during tactical engagements, 
has applications in the selection, initial and continuing training, and performance 
evaluation of the AO. It is hoped that issues raised and conclusions drawn in the course 



74 



of this experiment will stimulate further research in the application of human factors 
principles to modem submarine design. 
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APPENDIX A: TADMUS 



TADMUS 1 , which is currently in its second iteration of development, is entering 
the human factors test and evaluation phase. The program managers, after a full testing 
of the first iteration, significantly changed the layout, design, and cognitive models that 
drive TADMUS. Figure Al. 1 shows the current screen layout of TADMUS. TADMUS 
consists six distinct modules each with a specific purpose: Geo-Plot, Multi-CRO Access 
Panel, Track Profile, Response Manager, Track Summary, and Basis for Assessment. 




Fig Al.l TADMUS Display. 

Geo-Plot 

The Geo-Plot module is the primary display for the command team (see Figure 
A1.2). It uses standard Naval Tactical Data Exchange (NTDS) color symbology as well 
as system track numbers to code all air, surface, and subsurface elements in the 
battlespace. The map can be expanded or enhanced to show any level of clarity available 



1 The following discussion has been excerpted from Dr. Hutchins writings on TADMUS in 1995/1996. 
Hutchins, S.G. (1996). Principles for Intelligent Decision Aiding . (SPAWARTM 1718). DON: Coronado, 
CA. 

Hutchins, S.G. &Rummel, D.K (1995). A Decision Support System for Tactical Decision-making Under 
Stress. Proceedings of the first International symposium on Command and Control Research and 
Technology. Montery, CA. 
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in Joint Operational Tactical Exchange (JOTS). This ranges from general area maps to 
specific city street maps. In addition, users can overlay maps to increase detail or “de- 
saturate” them to reduce clutter. The Geo-Plot can also show weapons information such 
as range brackets, areas of uncertainty (AOU), and projected tracks (incoming and 
outgoing) to allow own ship to rapidly assess the criticality of a threat. 




Fig A1.2 Geo-Plot Display. 



Multi-CRO Access Panel 

The Multiple-Character Read Out (CRO) Access Panel module displays all of the 
relevant information concerning target identification and kinematics (see Figure A1.3). 
In addition, it allows for the most critical tracks to be monitored without additional 
interaction from the user with the system. The buttons are all identical with priority 
assigned from left to right in order of decreasing importance. The CRO’s are color coded 
to show threat status and “blink” when pertinent track information is available. 
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Fig A1.3 Multi-CRO Display. 

Track Profile 

The Track Profile module complements the Geo-Plot by displaying a side view of 
track altitude and range from own ship (see Figure A1 .4). Information is available as to 
where the track is, what the track has been doing since tagging, when the track is within 
its presumed weapons range, and when own ship is within weapons range. The module 
also allows for own ship to evaluate multiple weapons type responses as well as 
projecting weapons effectiveness sphere's from other friendly assets in the battlespace. 
This feature will be critical for the notion of future “Joint Battle-Groups”. 




Fig A1.4 Track Profile Display. 



Response Manager 

The Response Manager module provides a Gantt chart type display showing a set 
of pre-planned actions as well as the optimal range for execution for the target of interest 
(see Figure A1.5). These pre-planned actions are intended to be based on the current 
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Order of Battle that the commander is operating under and can be updated as frequently 
as required. 




Fig A1.5 Response Manager Display. 



Track Summary 

The Track Summary module supplements the Multi-CRO panel by providing a 
detailed summary of all current and historical classification and kinematic data for the 
selected track (see Figure A1.6). It also provides for the display of ancillary data, such as 
intelligence. Electronic Warfare (EW), and Identification Friend or Foe (IFF). 




Fig A1.6 Track Summary Display. 
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Basis for Assessment 



The Basis for Assessment module provides the command team with detailed lists 
of evidence for and against the current assessment of the selected track (see Figure A1.7). 
It also presents presumed, but yet unknown, information and implicit assumptions being 
made in accepting the classification of the contact as a potential non-threat, threat, or 
unknown. 




Fig A1.7 Basis for Assessment Display. 
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APPENDIX B: EXPERIMENTAL PROTOCOL AND INFORMED CONSENT 



EXPERIMENTAL PROTOCOL 



Ned Experimental Protocol (total duration ~2 — 3 hours) 

Conventions 

“=>“ is used before actions 

| Things in this format are to be read aloud verbatim . 

| Things in this format are to be read aloud verbatim and stressed 

Things in this format are notes to the experimenter 
Experiment Setup 

=> Put a fresh tape in the VCR and audio recorder, batteries in the audio recorder, 
and make sure the connections are all working 

=> Be sure to disable any applications that run on a scheduled basis, such as Norton 
FileSaver, mail programs, etc., these will disrupt the timing in Ned. 

=> Have the tic-tac-toe HyperCard stack up, running, and hidden. 

=> Have the TTT video movie up, running, and hidden. 

=> Have Ned (vl.5.5) up, running, and hidden. 

=> Be sure to have the informed consent forms handy 

Experiment Briefing (5-10 min.) 

The basic idea here is to give the subject a brief overview of the purpose of the study and 
what will be expected of them. 

=> Read: 

This is an unclassified system being used for basic research. This exercise will be 
audio/video recorded. Please keep all remarks unclassified. 

Today I will ask you to participate in two scenarios using a simplified command 
workstation simulation. The purpose of this experiment is to gather data on how 
command might use information that is typically available in the Combat Control 
System. As you know, the new SSN that is being planned will have a dedicated command 
workstation (CWS) to support the Approach Officer. As this is a new console, there is 
little known about how such a workstation will be used. This experiment is one in a series 
of studies to predict the answer to that question. One thing we do know, gleaned from the 
experience of many submarine officers, is that the CWS will require different displays 
than those currently available on operator consoles. Your participation will help 
determine the content of those displays. 

In this exercise, your role will be that of the AO. 
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Data collection in this experiment depends on making your information needs and usage 
visible. A large part of our task is to understand what information you need and when you 
need it. Hence, I will ask you to talk aloud as you work. I will explain this in more detail 
and give you an opportunity to practice doing this shortly. Videotapes will be made of 
every scenario and computerized data is being collected as well. 

In accordance with requirements ofNUWC's Committee for the Protection of Human 
Subjects, strict procedures have been established to preserve the confidentiality of 
participants. Outside of the experimenters, no one can gain access to information that 
would reveal your identity. If you have any concerns that I should address, please ask 
them now. 

=> Hand them the Informed Consent forms and ask them to read and sign 

Talk-aloud training (5-10 min.) 

The idea here is to give the subject’s some specific instructions on what sorts of things 
we are going to want them to report and also to get them comfortable with talking aloud 
while working. To this end, the subject is given instructions ala Ericsson & Simon and a 
practice task to work on with tic-tac-toe. The subject plays a couple of games and is 
given feedback on the quality of the protocols 
=> Bring up the tic-tac-toe game and TTT video 

=> Read: 

As you recall, I said that you are to talk aloud during this study. This may require some 
example and practice, so lets first watch the computer to it and then you’ll play the 
computer. 

=> Play TTT for one approximately one minute. 

=> Read: 

To reiterate, what I mean by talk aloud is that I want you to say out loud everything that 
you say to yourself silently. Just act as if you are alone in the room speaking to yourself. 
If you are silent for any length of time, I will remind you to keep talking aloud. Do you 
understand what I want you to do? 

You may begin playing when you are ready 

=> If silent for more than 10-15 seconds say “keep talking” 

=> Give feedback after each game 

=> Continue until subject does not need to be reminded more than once during the 
entire game. 

=> Quit HyperCard and MoviePlayer. 

=> You should be at the 15-minute point 

Introduction to Ned (25 - 30 min.) 

Freeplav Mode 

The goal of this part is to provide the subjects with enough knowledge about the interface 
to get good, clean data from the experiment sessions. Also, because data collection will 
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occur with information covered by the gray boxes, we can avoid extraneous mouse clicks 
by ensuring that subjects know where to find the information they are seeking. 

Prompted Practice 

=> Bring up Ned and read: 

I will now show you the simulated combat control system (CCS). With it you can 
monitor all normal command functions, ship's control, Sonar, and TMA (MATE, Geo Sit, 
and plots). You can also order changes in own ship's speed, course, and depth. As this is a 
simulation designed for experimental purposes and unclassified, there are some 
limitations and unrealistic elements, which I will note as we go along. 

j I will introduce this simulation in two phases. First I will walk you through all of the 
displays and give you an opportunity to ask questions about the contents of the displays 
and to practice using the interface controls. I will then switch the simulation over to data 
collection mode, which is the mode in which you will be working for the main part of the 
study, to let you practice and explore the simulation under those conditions. 

Please feel free to ask questions as we work through the displays, I want to make sure 
you understand how all of thee displays work before we start collecting actual data. 

I also want to remind you that this is an unclassified system being used for basic research. 

This exercise will be audio/video recorded so please keep all remarks unclassified. 

=> Make sure the Practice scenario is selected in the pop-up menu 

=> Press the “Start” button 

=> Let the subject have control of the computer 

=> Ned will display the “Problem Description” and then “Status Report” windows. 
Read the text on these displays aloud and press the buttons to continue. 

=> When it gets to OS control, read the following and point to each thing as you 
describe it. 

This window allows you to control own ship. These fields contain the ordered course, 
speed, and depth. To change course, select the field, type the new ordered course, select 
the direction of the turn, then click on this initiate change button. These fields above 
display the current actual, course speed and depth. 

Go ahead and turn right to a bearing of 150, and slow down to 7 knots. 

While OS is maneuvering, read: 

| Note that the fields up top change as OS changes, while the fields below remain constant. 

=> When maneuver is complete, read: 

This is the display navigation palette. You will be able to look only at one display at a 
time. To look at a display, simply press the appropriate button. We will first go to the 
broadband spherical waterfall display. Go ahead and click the button labeled “BB- 
Sphere”. 

=> When BB-Sphere comes up, read: 
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This is the waterfall window for the broadband sphere. It displays true bearing from 0 to 
359 degrees and is divided into two parts. In the upper part the integration time is twenty 
seconds. In the lower part it is ten minutes. OS course is represented by this blue line in 
the upper part. 

As you can see we have a trace here at a bearing of about 220, which corresponds to the 
merchant contact from our status report. To put a tracker on a target, just position the 
cursor over the track until it changes shape, then click. Go ahead and set a tracker on this 
target. 

=> When tracker is set, read: 

As you can see, the information on the track is displayed here. It assigns a sierra number, 
in this case sierra 1, and gives you the true bearing, SNR, and DE. 

Along with a broadband sphere, you also have a broadband towed sonar sensor. If you 
don’t have any questions about this display, go ahead and click the button labeled “BB- 
Towed”. 

=> When BB-Towed comes up, read: 

Like the spherical sensor, the broadband towed display is also a waterfall display with 20 
seconds of integration time in the top part and 10 minutes of integration time in the 
bottom part. Unlike the bb-sphere, this display plots conical angle from 0 to 180 degrees. 
You set a tracker in the same manner as in the other display - just move the cursor over 
the trace until it changes, and then click. Go ahead and set a tracker on this trace. 

=> When tracker is set, read: 

As before, the information on the track is displayed here. It assigns a sierra number, in 
this case sierra 1, and gives you the conical angle, both ambiguous bearings and SNR. In 
this field below, you can check the status of the towed array. As you can see, the array is 
currently stable. 

The third sonar sensor available to you is narrowband towed. If you don’t have any 
questions about this display, go ahead and click the button labeled “NB -Towed”. 

=> When NB-Towed comes up, read: 

This display shows frequency on the horizontal axis and conical angle on the vertical 
axis. The frequencies displayed here are completely fabricated due to the unclassified 
nature of the simulation. The key point is that frequencies to the left are where you are 
likely to find a merchant contact and frequencies to the right are where you would find a 
submerged contact. As you can see, we have one of each currently on sonar. Go ahead 
and set trackers on both of these targets. 

=> When tracker is set, read: 

As you can see, the information on the tracks is displayed off to the right, and includes 
sierra number, conical angle, both ambiguous bearings and SNR. The status of the towed 
array is shown below. 
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Data from these three sensors are sent to several other displays, including a time bearing 
plot, time frequency plot, and TMA. Lets start with time bearing, go ahead and press the 
TimeBmg button on the palette. 

=> When TBR comes up, read: 

As you can see, this display plots 0 to 360 degrees of bearing over a 20 minute period. 

To use this display, first select the sensor for which you would like to see data. Go ahead 
and click the boxes labeled bb-spherical and nb-towed. 

=> When boxes are checked, read: 

The traces correspond in color to the label on the boxes, and the size of the boxes 
correspond to target. The larger boxes are Sierra 1 and the smaller ones are Sierra 2. To 
get the bearing rate, move the magnifying glass over the box of interest and read the 
bearing rate from the field below. Note that true bearing is also available. Each box 
represents 20 seconds of data, so the bearing and bearing rates you are reading have been 
averaged over 20 seconds. 

The narrowband sensors send data to the time frequency plot, which we will look at next. 
If you don’t have any questions, go ahead and press the TimeFreq button on the palette. 

=> When TFP comes up, read: 

This display plots a 40 hertz span of frequency over a 10 minute period. Each box 
represents data averaged over 6 seconds. This display works in a manner very similar to 
the time bearing window. First you select a target from the pop-up menu over here, then 
you move the magnifying glass over the box to read the frequency. The time that is 
displayed is the scenario time that the box was plotted. 

If you have no questions about this display, we will move on to the target motion analysis 
window. Go ahead and click the TMA button. 

=> When TMA comes up, read: 

This display allows you to work one solution at a time for each target and sensor. You 
select the target/sensor from this pop-up menu. Go ahead and select the broadband 
spherical solution for sierra 1. 

=> When sierral-bbs solution appears, read: 

As you can see, the system has provided you with a starting solution. This solution is 
displayed in the solution data box. It assigns the solution an ID#, and displays the range, 
bearing, course, speed, bearing rate, and range rate, as well as the age of the tracker 
sending data to TMA. 

This solution is plotted on the geosit below. You have a choice of four zoom levels via 
this pop-up menu. The number on the menu corresponds to the outermost circle on the 
geosit. Go ahead and select the 40 kyd scale. The target is plotted as a circle with a course 
speed vector. A bearing line is plotted for the spherical array, and a parabola for the 
towed arrays. 

Over to the left are plots of the solution bearing and the error stack. These can plot up to 
15 minutes of data at a time. 
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| To change the solution, press the tweak solution button. Go ahead and press it now. 

=> When tweak solution dialog appears, read: 

In this dialog, you can modify the solution range, bearing, course and speed. To change 
parameters, simply type the new values in the fields and press either the preview or the 
done button. The preview button will show the changes in the TMA window, but keep 
the tweak solution dialog open. The done button will make the changes and close the 
dialog. Go ahead and change a few parameters and then hit the preview button. 

=> When new solution displayed, read: 

As you can see, the new solution appears in all parts of the time window. If you choose 
to keep the solution, press the done button. If you want to modify it further, you can type 
new parameters in the fields and then hit preview or done. If you want to go back to the 
solution you had before you hit the tweak solution button, press cancel. 

Note that whenever the tweak solution dialog is open, you will be unable to do anything 
else in the window, so be sure to close the dialog, either by hitting cancel or done, before 
attempting to do anything else. Go ahead and hit the done button now. 

If there are no questions on the TMA window, lets move on to the Geosit window. Go 
ahead and click the GEOSIT button now. 

=> When GEOSIT conies up, read: 

This display will allow you to see solution plots on both targets simultaneously. To view 
a solution, simply select it from one of these two pop-up menus. The top, black menu is 
for sierra 1 and the lower, red menu is for sierra 2. As you can see, there is also a menu to 
select the scale of the plots, again, with the number on the menu corresponding to the 
outermost ring. 

Go ahead and select the broad band spherical solution for sierra 1 and the narrowband 
towed solution for sierra 2. 

=> When solutions displayed, read: 

As you can see, each target is plotted as a circle with a course speed vector. The colors 
of the targets correspond to the colors on the pop-up menus, with sierra 1 in black and 
sierra 2 in red. Note that this display also plots bearing lines. It will plot a maximum of 5 
bearing lines each 2 minutes apart. 

If there are no questions on this display, let’s move on to the line-of-sight display. Go 
ahead and hit the LOS button now. 

=> When LOS comes up, read: 

This is a passive display in that it has no data being sent to it. You will have to provide 
the values. In order to use it, enter values into each of these fields and then press the plot 
button. Go ahead and enter in some values now. Don’t worry it they are correct or not 
for now. 

=> When LOS plotted, read: 

|| As you can see, the target’s course is plotted above and ownship’s is below. 
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If there are no questions about this display, let’s move on to the final display, the sound 
velocity profile. Go ahead and hit the SVP button now. 

=> When LOS comes up, read: 

This display is inanimate and simply plots the current SVP. There are no actions that can 
be performed on this display. 

Do you have any questions on this or any other display? 

Okay, then we will move on to the next phase of the study where I show you some of the 
features of the simulation in its experiment mode. Go ahead and click the firing point 
procedures button to stop the simulation. 

=> Take control of the computer. When the main control window comes up click 
“Stop” then “Reset”. 

Experiment Mode 

The goal here is to familiarize the subject with the experiment-specific stuff - primarily 
using the gray boxes and talking aloud while working. It is important to describe in some 
detail the use of the boxes in the TBR and TFP, as they work a little differently than they 
do in other displays. 

After the demonstration, the subject is instructed to try to determine a solution for the 
scenario and to talk aloud while doing so. Provide feedback on the quality of the 
protocols. When the subject is providing good quality protocols and is comfortable with 
the use of the gray boxes he is ready for the data collection phase. 

=> Select the “Run experiment” check-box and press the “Configure...” button. 

=> Enter your initials, the subject number, and trial number of “0” in the Setup 
window 

=> Press the “Create Data File...” button. 

=> Make sure the “Collect Biographical Data” check box is NOT checked and the 
“Wash-up” one is. Press the “Done” button when finished. 

=> Press the “Start” button to bring up the problem description window 

=> Let the subject have control of the computer 

Prompted practice 
=> Read: 

Now we are going to look at experiment mode. There will be a few differences from what 
I just showed you. We are running the same practice problem that we did the last time so 
the problem description and scenario description are the same. Just go ahead and press 
the View Status report and Begin buttons on the first two windows. 

=> When OSC window comes up, read: 

One major difference is that all data fields are covered by gray boxes until you put the 
cursor in them and click the mouse button. These fields are exactly the same as the ones 
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you saw before, they are just covered up by the boxes. Go ahead and click on the box 
over ordered OS course. 

=> When OS course field is visible, read: 

When you move the mouse out of the field, the information is covered back up. Go ahead 
and move the mouse out of the field. 

=> When OS course field covered back up, read: 

Some of the boxes are very small, so try not to move the mouse out unintentionally. 

The fields do receive data, but the values are not visible until you click on them. 
Likewise, you also won’t be able to enter a value into a field unless it is visible. 

Generally speaking, you will be able to look at only one thing at a time. 

Go ahead and turn to the right to 150 degrees, and then watch the values change in the 
course status field above. 

=> When OS has completed the maneuver, read: 

Okay, lets go to the narrowband towed display so you can practice setting a tracker in 
experiment mode. 

=> When NB-Towed comes up, read: 

You won’t be able to set a tracker unless you uncover the waterfall display first. Go 
ahead and click on the waterfall box and set a tracker on the merchant. 

=> When tracker has been set, read: 

As you can see, the information on the tracker is also covered up. To see the conical 
angle, bearings, or SNR, you will have to click on the appropriate gray box. Go ahead 
and click on these. 

The fields in the displays all work the same as this except for the Time Bearing and Time 
Frequency displays. Let’s go to the time bearing display first. 

=i> When TBR comes up, read: 

| As before you will first have to select a sensor. Go ahead and click nb-towed. 

=> When narrowband-towed data plotted, read: 

You also have to choose whether you want to see bearing or bearing rate. You will only 
be able to see one at a time. Go ahead and click on the box next to bearing rate. 

=> When bearing rate box uncovered, read: 

Unlike the other gray boxes, this one will stay open when the mouse leaves. This is so 
you can move the magnifying glass around to get information from the data plotted on 
screen. 

Go ahead and move the magnifier over some data. 

=> When subject has done this, read: 
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This box will stay open until you select bearing. Go ahead and click on the box next to 
true bearing. 

=> When bearing box uncovered, read: 

As you can see, bearing rate is now covered, so you can only see the true bearing. 

The TFP display works very much like this. Let’s take a look at that one now. 

=> When TFP comes up, read: 

The frequency and time fields act in the same manner as bearing rate and true bearing 
fields in the TBR window. Go ahead and select S001 from the pop-up menu and then 
click the box next to freq. 

=> When subject has done this, read: 

Now when you move the magnifying glass over the data, you can read the frequency but 
not the time. 

Free Practice 
=> Read: 

Now it is your turn to practice using the simulation in this mode. Feel free to ask 
questions as you work. I would also like you to practice talking aloud as you do this. 
When you are comfortable with the system and feel you have a reasonable solution on the 
targets, select this button labeled FPP. It stops the scenario and takes you to a display for 
a hot wash-up. 

=> Watch the subject interact with the system, answering any questions he has 
Wash-up Prompted Practice 

=> When the GEO DRAW window appears, read: 

Here we will want you to indicate what happened during the scenario. If there were 
discrepancies between your solution and the system, you should indicate your solution 
rather than the system solution. 

Here is how it works. First you select a contact, in this case, Sierra 1 A is already selected 
for you. So go ahead and click in on the geosit the bearing and range where you think the 
merchant was at the end of the leg. 

=> When the subject has dropped the target, read: 

The next mouse click draws a course-speed vector from the center of the target to 
wherever you click. Go ahead and do this now. 

=> When the subject has drawn the vector, read: 

A third mouse click draws a line indicating of range uncertainty surrounding the target. 
Click anywhere along the vector from the center of the geosit to the target. 

=> When the subject has drawn range uncertainty, read: 

| As you can see, this draws a gray line surrounding the target. 
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You may indicate bearing ambiguity by selecting a second version of the same contact 
number, using, for example, 1 A for the more likely ambiguous bearing and IB for the 
other ambiguous bearing. Select sierra lb from the pop up menu and draw the ambiguous 
bearing on the merchant, including a course speed vector and range uncertainty. Don’t 
worry if it is correct or not, this is only practice. 

=> When the target has drawn sierra IB on the geosit, read: 

If you want to delete a target, select the target from the pop-up menu and press the 
“Remove” button. Do this with sierra 1 A. You will have to select it from the pop-up 
menu first. 

=> When SierralA has been deleted, read: 

Also indicate own ship course by selecting ownship from the menu and clicking to 
represent the appropriate course vector. Go ahead and draw ownship vector now. 

=> When OS vector has been drawn, read 

There are instructions on the window to remind you how to do all of this. 

When you are satisfied, click the “Done” button. Go ahead and do this. 

=> When GEO TABLE window comes up, read 

You indicate your best guess for target course, speed, and range estimates on this page. 
You just type values into the appropriate fields. 

I want you to provide information on both of these wash-up displays for each leg of the 
scenario, so if there is a leg you haven’t provided information on yet, press this “Next 
Leg...” button. If have provided information on all of the legs of the scenario, press this 
“Done” button. This will end the exercise. 

Wash-up Free Practice 

At this point, I would like you to practice using both of these displays without me 
stepping you through it, so go ahead and press the next leg button and draw the last leg of 
the scenario. 

=> When the GEO DRAW window appears, read: 

Just go ahead and place the targets from the practice scenario on the geosit, making sure 
to include the course/speed vector and the range uncertainty. Also plot the course of 
OwnShip. Try using the “Remove” button to make sure you understand how it works. 
When you finish, press the “Done” button to get to the table screen. On the table screen, 
enter numerical values to represent what you drew on the graphical screen and hit the 
“Done” button when you are finished. If you have any questions while doing this, please 
ask. 

=> Watch the subject interact with the screens and answer any questions they have 

=> When they press the “Done” button and get back to the main control screen, take 
control of the computer and press the “Stop”, then the “Reset” button 

=> You should be at the 45-minute point 
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Session 1 data collection (30 min.) 

The subject is provided with the same cover story as the last time (i.e., Russia and 
Ukraine may be out to attack merchant shipping), and also given the specific status report 
for the first scenario - to include OS course, speed, depth, MERC bearing. The subject 
interacts with Ned, uninterrupted, until he feels confident enough in the solution to fire, 
or 30 minutes have passed. The experimenter plays a passive role, only prompting the 
subject to talk aloud when there is a > 10 second or so period of silence. Subjects will 
complete both the graphical and tabular forms of the hot wash-up 

=> Start the video tape recording 
=> Read: 

| If you have no further questions, we will start the first real scenario. 

=> Load the desired scenario by selecting it from the pop-up menu. 

=> Make sure the “Run experiment” check-box is still checked and press the 
“Configure...” button. 

=> Enter the appropriate values in the setup window, naming the data file according 
to the suggestion in the save file dialog. Make sure the “Collect Biographical 
Data” check box is NOT checked and the “Wash-up” one is. Press the “Done” 
button when finished. 

=> Press the “Start” button to bring up the problem description window 
=> Read: 

All scenarios take place with this same basic problem description that you saw in 
practice. We will begin with detection and run until the point at which you would like to 
initiate firing point procedures. You should indicate this by pressing the FPP button on 
the display navigation palette. You will then do the hot wash-up as you did in the 
practice scenario. 

Please talk aloud during the entire problem - just sav out loud everything that you sav to 
yourself silently. 

You may begin when ready. 

=> Watch passively as subjects complete scenario. Interrupt only if they are silent 
for 10-15 seconds or so, when you do so, just say “talk aloud”. 

=> When subject has completed the scenario, and gotten through the hot wash-up, 
take over the computer and press the “Stop”, then “Reset” button. 

=> See if the subject needs to take a short break 

=> You should be at the 1:15 minute point. 

Session 2 data collection (30 min.) 

Same as Session 1 except that subjects work a different scenario, and biographical data is 

collected 

=> Read: 

| We will now start the next scenario. 
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=> Load the desired scenario by selecting it from the pop-up menu. 

=> Make sure the “Run experiment” check-box is still checked and press the 
“Configure...” button. 

=> Enter the appropriate values in the setup window, naming the data file according 
to the suggestion in the save file dialog. Make sure the “Collect Biographical 
Data” AND “Wash-up” check boxes are both checked. Press the “Done” button 
when finished. 

=> Press the “Start” button to bring up the problem description window 
=> Read: 

Recall that all scenarios take place under the same basic conditions. As before, we will 
begin with detection and run until you decide you are ready to go to firing point 
procedures. After the hot wash-up, you will also be shown a screen containing some 
biographical questions. 

Again. Please talk aloud during the entire problem 
You may begin when ready. 

=> Watch passively as subjects complete scenario. Interrupt only if they are silent 
for 10-15 seconds or so, when you do so, just say “talk aloud”. 

=> When subject has completed the scenario, and gotten through the hot wash-up, 
and biographical data screens, take over the computer and press the “Stop”, 
then “Reset” button. 

=> See if the subject needs to take a short break 
=> You should be at the 1:45 minute point. 

Experiment debrief and form-filling (15-25 min.) 

The goal here is to examine recall of both scenarios and the distinctiveness of the recall. 
These should be done with paper and pencil and encourage the subjects to explain as they 
go. It might also be interesting to collect data on what the subjects made of the simulation 
(reality, usability, etc.) after they complete the recall session. This can be rather informal. 
Scenario 1 Recall 

=> Load the practice scenario by selecting it from the pop-up menu. 

=> Make sure the “Run experiment” check-box is still checked and press the 
“Configure...” button. 

=> Enter the appropriate values in the setup window, entering 1W as the trial 
number for the first scenario 

=> Name the data file according to the suggestion in the save file dialog. Make sure 
the “Collect Biographical Data” check box is not checked and the “Wash-up” 
one is. Press the “Done” button when finished. 

=> Press the “Start” button to bring up the problem description window 
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=> Press the view status report and begin buttons as they come up, and then press 
the FPP button on the control window when it appears to get to the GEO DRAW 
window 

=> Read: 

This is the last portion of the experiment. I would like you to use both the graphical and 
tabular wash-up screens to draw the events of both scenarios as you did before. Begin 
with the first scenario, drawing the targets on the graphical window and then entering 
numerical values in the tabular window. Be sure to do this for all targets and ownship 
and for each leg. You may begin when ready. When you are finished, hit the Done 
button on the tabular window. 

=> When subject has finished take over the computer and press “Stop”, then the 
“Reset” button. 

Scenario 2 Recall 

=> Load the practice scenario by selecting it from the pop-up menu. 

=> Make sure the “Run experiment” check-box is still checked and press the 
“Configure...” button. 

=> Enter the appropriate values in the setup window, entering 2W as the trial 
number for the first scenario 

=> Name the data file according to the suggestion in the save file dialog. Make sure 
the “Collect Biographical Data” check box is NOT checked and the “Wash-up” 
one is. Press the “Done” button when finished. 

=> Press the “Start” button to bring up the problem description window 

=> Press the view status report and begin buttons as they come up, and then press 
the FPP button on the control window when it appears to get to the GEO DRAW 
window 

=> Read: 

I would now like you to recount the events in the second scenario. Be sure to do this for 
all targets and ownship and for each leg. You may begin when ready. When you are 
finished, hit the Done button on the tabular window. 

=> When subject has finished take over the computer and press “Stop”, then the 
“Reset” button. 

Final Questions 

=> Transfer all files to zip disk and label appropriately 
=> Hand the subject the additional questionnaire. 

=> Stop Video Camera and Audio Recorder. 

Please take a minute to fill out the additional questionnaire. 
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INFORMED CONSENT 



I have been informed about the purposes and procedures of this experiment 
according to enclosure (1) and am willing to participate under the conditions 
stated. I understand that: 

(a) I may withdraw at any time with no penalty, 

(b) All data will be confidential, and 

(c) All data will be retained by the Principle Investigator named in 
enclosure (2). 



Print Name: 
Signature:_ 
Date: 
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GENERAL INFORMATION 



This is an unclassified system being used for basic research. This 
exercise will be audio/video recorded. Please keep all remarks 
unclassified. 

Today we will ask you to participate in two scenarios using a simplified simulation. 
First we will provide a high level overview of what you will be doing and why we are 
asking you to do it. When we finish we will tell you more specifically what we are 
studying and how your participation contributes to the goals of the study. 

Purpose 

The purpose of this experiment is to gather data on how command might use 
information that is typically available in the Combat Control System. As you know, the 
new SSN that is being planned will have a dedicated command workstation (CWS) to 
support the Approach Officer. As this is a new console, there is little known about how 
such a workstation will be used. This experiment is one in a series of studies to predict 
the answer to that question. One thing we do know, gleaned from the experience of many 
submarine officers, is that the CWS will require different displays than those currently 
available on operator consoles. Your participation will help determine the content of 
those displays. 

Overview of the study 

Your role will be that of the AO. Each scenario will begin with a status report on the 
current situation. Before the first scenario begins, you will be given practice with the 
experimental procedures and with the simulation. 

Data collection in this experiment depends on making your information needs and 
usage visible. A large part of our task is to understand what information you need and 
when you need it. Hence, we request that you think aloud, saying whatever comes to 
mind, even if it might be irrelevant. You will be given an opportunity to practice doing 
this shortly. Videotapes will be made of every scenario and much computerized data is 
being collected as well. In accordance with requirements of NUWC's Committee for the 
Protection of Human Subjects, strict procedures have been established to preserve the 
confidentiality of participants. Outside of the experimenters, no one can gain access to 
information that would reveal your identity. If you have any concerns that we should 
address, please ask them now. 
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STUDY INFORMATION 



Title of the Study: 

Project NEMO 

Principal Investigator (Name, Code, Phone): 

Susan S. Kirschenbaum, Ph.D. Code 2211, 841-4381, x23835. 

Sponsors: 

Dr. S. Dickinson, NUWC IR Program Manager, and Dr. S. Chipman, Code 342, 
ONR. 

The purpose of this work is to study and model the cognitive activity and 
behavior required for situation assessment. The nominal subject will be the 
submarine Approach Officer (AO). Situation assessment is a complex cognitive 
skill involving components (such as information search and assessment, memory, 
problem solving, and decision making) that have been studied in isolation, but 
never in the context of event-driven situation assessment. We expect our pursuit of 
situation assessment to result in advances in both theory and practice. 

This experiment will take place in the NUWC Division Newport laboratory 
facility, the ARCH Laboratory at George Mason University, or at the participant’s 
own facility. It will consist of subjects interacting with staged scenarios using a 
simplified simulation created for this experiment and hosted on a Macintosh. 
Subjects will be doing a task that is similar to their normal job including assessing 
data and giving orders for the control of the simulated submarine in a specified 
mission. They will each have two scenario segments. The total experimental time 
will be approximately two hours. All sessions will be video taped 

This experiment is expected to require 10 to 15 subjects in each of three 
groups, novices (prior to first deployment), intermediates (after two to four years at 
sea), and experienced (Executive Officer or more experience). Subjects will be run 
individually, at the convenience of the subjects. No deception will be employed. 
The standard informed consent form will be used with a description of the 
experiment. Data recording will include computer interactions, videotapes of the 
screen, audio recordings of think-aloud protocols, and written questionnaires. All 
records will be retained by the P.I. No data will be identified by subject. Data will 
be analyzed by statistical and process-tracing methods, including cognitive 
modeling and other appropriate methods. Subjects will be able to request copies of 
resulting publications and will be given time at the end of sessions to ask 
questions. 

The P.I. of this project is an experimental psychologist and agrees to conduct 
the experiment under conditions recommended by the American Psychological 
Association and approved by this committee. 
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APPENDIX C: EXPERIMENT KEY 
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Table C4.2. Timeline for 60 Minute Scenarios (course and speed remain the same as initial values above unless specified). 




zl becomes High Dby 
on right, drawing rt 
Course, speed, By, Range 
OS 000, 9 

Merch 334, 12, 113, 15000 
Sub 184,10,043, 7000 




Figure C4.2. Scenario Zl Geometry. 
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New Z2, High Dby 
on left, drawing left 
Course, speed, By, range 
OS 225, 9 
Merch 160, 12, 160,15000 
Sub 350, 10, 150, 7000 




Figure C4.4 Scenario Z2 Geometry. 
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Trial Order 
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Table C4.3 Latin Square Design. 
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APPENDIX D: ADDITIONAL QUESTIONAIRRE 



ADDITIONAL QUESTIONNAIRE 

DATE: 

INTERVIEWER: 

SUBJECT : 

1 . Could you tell me what you thought of the situation? 

2. Was the situation “realistic” enough for you? Any comments? 

1 Very Unrealistic 

2 Unrealistic 

3 Borderline 

4 Realistic 

5 Very Realistic 

3. How difficult were the scenarios? (Please provide a score for each run) Any 
comments? 

1 Very Difficult 

2 Difficult 

3 Borderline 

4 Easy 

5 Very Easy 

4. Please rate the effectiveness of each display. 

1 Very Ineffective 

2 Ineffective 

3 Borderline 

4 Effective 

5 Very Effective 

LOS OSC SVP 

TMA TAPBB TAPNB SAPBB 

TF TB GEOSIT 



5. Please rank the usefulness of the displays. One for the most useful, 10 for the least. 
No ties allowed. 

LOS OSC SVP 

IMA TAPBB TAPNB SAPBB 

TF iTB GEOSIT 
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6. The ultimate goal of the project is to provide for the design of the CWS of the NSSN. 
If you were in charge, what kind of design would you make? Include any and all 
displays/information/graphical interfaces that you desire. 

(Use Blank sheets provided) 

7. Any more comments? 
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APPENDIX E: ASSUMED TORPEDO CHARACTERISTICS 



Name: 

Manufacturer: 

Diameter: 

Length: 

Weight 

Range: 

Speed: 

Operating Depth: 
Acoustic acquire range: 

(Friedman, 1983) 


MK-48 Heavyweight Torpedo 

Gould Electronics 

21 inch 

230 inch 

3480 pounds 

35000 yards 

55 knots 

2500 feet 

2000 yards 
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APPENDIX F: VB MARCO CODE 



Option Base 1 

Sub doall { ) 

Call breakoutdata 
Call displayanalysis 
End Sub 

Sub breakoutdata ( ) 

Dim newdata, newtruth, datcount, truthcount 
datcount = 0 
truthcount = 0 

Set newdata = ActiveWorkbook. Worksheets .Add 
Set newtruth = ActiveWorkbook. Worksheets .Add 
newdata. Name = "Data" 
newtruth. Name = "Truth" 

For Each rw In Sheets { 3 ). Rows 

If rw. Cells (1) = "" And rw.Row > 1 Then 
Exit For 
Else 

If rw. Cells (1) = ("TRU") Then 

truthcount = truthcount + 1 
newtruth. Rows (truthcount) .Value = rw. Value 
Elself rw.Row > 6 Then 

datcount = datcount + 1 

newdata. Rows (datcount) .Value = rw. Value 

End If 
End If 
Next rw 

Call cleantruth (newtruth) 

Call cleandata (newdata) 

End Sub 

Sub cleantruth (truthsheet) 

For N = 1 To 5 

truthsheet . Columns (1) .Delete 
Next N 

truthsheet . Columns (2) .Delete 

truthsheet. Columns (1) . NumberFormat = "h:mm:ss" 

End Sub 

Sub cleandata (datasheet) 

For N = 8 To 20 

datasheet . Columns (8) .Delete 
Next N 

With datasheet 

.Columns (6) .NumberFormat = u h:mm:ss" 

.Rows (1) .Font. Bold = True 
. Columns ("A: BW") .AutoFit 

.Range ("A: BW") .Sort keyl: = . Columns ("E") , header : =xlYes 
End With 

datasheet .Activate 
ActiveWindow. SplitRow = 1 
ActiveWindow. FreezePanes = True 
End Sub 
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* Master function to run the analysis by display 
Sub displayanalysis ( ) 

Call displayaccess summary 
Call displaysequence 
Call addfreqchart 
Call addtotchart 
Call addavgchart 
Call repositioncharts 
End Sub 



stdevstr. 



Sub displayaccesssummary ( ) 

Dim countsheet As Variant 

Dim discount, rowcount, oldtime, newtime As Integer 
Dim fpctstr, tpctstr As String 'sumstr, avgstr. 

Dim labarray, disarray, timearrayO As Double 

disarray = Array ( "GEOSIT" , "LOS", "OSC", "SONAR-BB- SPHERE" , 

"SONAR-BB-TOWED", " SONAR-NB- TOWED " , "SVP", "TBR" , "TFP", "TMA" ) 
labarray = Array ( "Display" , "Frequency", "% Frequency", "Tot Time", 
"%Time", "Avg Time", "Std Dev") 
rowcount = 1 

Set countsheet = ActiveWorkbook. Worksheets .Add 
Call prepcountsheet ( countsheet, labarray) 

For Each d In disarray 
discount = 0 
oldtime = 0 
ReDim timearray(l) 

For Each rw In Worksheets ( "Data" ). Rows 
If rw. Cells (1) = "" Then 

rowcount = rowcount + 1 

'sumstr = "=sum(R" & rowcount & "C8" & " : R" & rowcount & 

"C" & UBound (timearray) + 6 & ")" 

'avgstr = "=average (R" & rowcount & "C8" & ":R" & rowcount 
& "C" & UBound ( timearray) + 6 & ")" 

'stdevstr = "=stdev(R" & rowcount . 

& "C" & UBound (timearray) + 6 & ") 
fpctstr = "=(R" & rowcount & "C2" . 
tpctstr = "=(R" & rowcount & "C4" « 

With countsheet 

. Range (. Cells ( rowcount, 8), . Cells ( rowcount, discount + 

7 ) ) . FormulaArray = timearray . Cells ( rowcount, 1) .Value 



'C8 " & " ; R" & rowcount 



»i j H 
•I j H 



"R12C2 ) 
"R12C4 ) 



= d . Cells ( rowcount. 



End 



.Cells (rowcount, 
.Cells (rowcount, 
. Cells ( rowcount, 
.Cells (rowcount, 
.Cells (rowcount, 
.Cells (rowcount, 
. Cells ( rowcount, 
. Cells ( rowcount. 
With 



2) .Value = discount 

3) . FormulaRICl = fpctstr 

4 ) . FormulaRlCl = sumstr 

5) . FormulaRICl = tpctstr 

6) . FormulaRICl = avgstr 

7) . FormulaRICl = stdevstr 
4) .Value = mysum ( timearray) 

6) .Value = myaverage ( timearray ) 

7) .Value = mystdev ( timearray) 



Exit For 

Elself rw. Cells (2) = "DISPLAY-CHANGE" And rw. Cells (3) = d Then 
discount = discount + 1 
newtime = rw. Cells ( 5) .Value 
ReDim Preserve timearray (UBound (timearray) + 1) 
timearray (discount ) = newtime - oldtime 
oldtime = newtime 
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Elself rw. Cells (2) = " DISPLAY- CHANGE" Then 
oldtime = rw. Cells ( 5) .Value 
End If 
Next rw 
Next d 
End Sub 

Sub prepcountsheet {sheet, labelarray) 

With sheet 

.Name = "Displays" 

. Range { "Al: Gl") . FormulaArray = labelarray 
.Range ("A1:G1") .Font. Bold = True 
.Cells {12, 1). Value = "Totals" 

.Cells (12, 1). Font. Bold = True 

.Cells (12, 2) .FormulaRICl = "=sum (R2C2 : R11C2 ) " 

.Cells (12, 4) .FormulaRICl = "=sum (R2C4 : R11C4 ) " 

. Range ( "Al : Gil" ). Borders . LineStyle = xlContinuous 
End With 
End Sub 

Function mysum ( anarray) 

For Each el In anarray 
mysum = mysum + el 
Next el 
End Function 

Function myaverage (anarray) 

If UBound ( anarray) > 1 Then 
myaverage = mysum ( anarray) / (UBound (anarray) - 1) 

Else 

myaverage = 0 
End If 

End Function 

Function mystdev (anarray) 

Dim sm, sqrd, N As Integer 
N = UBound (anarray) - 1 
For Each el In anarray 
sm = sm + el 
sqrd = sqrd + el A 2 
Next el 
If N > 1 Then 

mystdev = Sqr(((N * sqrd) - sm A 2 ) / (N * (N - 1) ) ) 

Else: mystdev = 0 
End If 

End Function 

Sub displaysequence ( ) 

Dim labarray, disarray, rowcount, oldtime, newtime 
disarray = Array ( "GEOSIT" , "LOS", "OSC", "SONAR-BB- SPHERE" , 
"SONAR-BB-TOWED", " SONAR-NB- TOWED " , "SVP", "TBR" , "TFP" , "TMA") 
labarray = Array ( "Time", "Display", "Duration") 
rowcount = 16 
For Each d In disarray 
oldtime = 0 

For Each rw In Worksheets ( "Data" ). Rows 
If rw. Cells (1) = "" Then 
Exit For 

Elself rw. Cells (2) = "DISPLAY- CHANGE" And rw. Cells (3) = d Then 
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newtime = rw. Cells (5 ) .Value 
rowcount = rowcount + 1 
With Worksheets ( "Displays" ) 

. Cells { rowcount, 1) .Value = newtime 
. Cells ( rowcount, 2) .Value = d 

. Cells (rowcount, 3) .Value = newtime - oldtime 
End With 

oldtime = newtime 

Elself rw. Cells (2) = "DISPLAY- CHANGE" Then 
oldtime = rw. Cells (5) .Value 
End If 
Next rw 
Next d 

With Worksheets ( "Displays" ) 

. Range ( "A15 : CIS" ). FormulaArray = labarray 

.Range ( "A15 : C15" ) .Font. Bold = True 

.Range ( "A16 : C1000" ) .Sort keyl :=. Columns ( "A" ) 

.Columns ("A: BW") .AutoFit 
End With 
End Sub 

Sub addf reqchart { ) 

Charts .Add 

ActiveChart.ChartType = xlColumnClustered 
ActiveChart . SetSourceData 

Source :=Sheets ( "Displays " ) .Range ( "A1 : All, Cl : Cll" ) , 

Plot By : =xlColumns 

ActiveChart . Location Where : =xlLocationAsOb j ect , Name : ="Displays " 
With ActiveChart 

. HasTitle = True 

. ChartTitle. Characters .Text = "Display Frequency" 

.Axes (xlCategory, xlPrimary) .HasTitle = False 
.Axes (xlValue, xlPrimary ). HasTitle = True 

.Axes (xlValue, xlPrimary) .AxisTitle . Characters . Text = "% 
Frequency" . HasLegend = False 
End With 
End Sub 

Sub addtotchart { ) 

Charts .Add 

ActiveChart.ChartType = xlColumnClustered 
ActiveChart . SetSourceData 

Source : =Sheets ( "Displays " ) .Range ( "A1 : All, El : Ell" ) , _ 

Plot By :=xlColumns 

ActiveChart . Location Where : =xlLocationAsOb j ect , Name : ="Displays" 
With ActiveChart 

.HasTitle = True 

. ChartTitle . Characters . Text = "Display Time" 

.Axes (xlCategory, xlPrimary) .HasTitle = False 
.Axes (xlValue, xlPrimary) .HasTitle = True 

.Axes (xlValue, xlPrimary ) .AxisTitle . Characters . Text = "% Time" 
.HasLegend = False 
End With 
End Sub 
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Sub addavgchart ( ) 

Charts .Add 

ActiveChart . ChartType = xlColumnClustered 
ActiveChart . SetSourceData 

Source : ^Sheets ( "Displays " ) . Range ( "A1 : All, FI : Fll" ) , _ 

PI ot By : =xl Columns 

ActiveChart . Location Where : =xlLocationAsObj ect , Name : = " Displays 
With ActiveChart 

. HasTitle = True 

. ChartTitle . Characters . Text = "Average Display Time" 

.Axes (xlCategory, xlPrimary) .HasTitle = False 
. Axes (xlValue, xlPrimary) . HasTitle = True 

. Axes (xlValue, xlPrimary) .AxisTitle . Characters . Text = "Time 
(secs) " .HasLegend = False 
End With 
End Sub 

Sub repositioncharts ( ) 

ActiveSheet . Shapes ( "Chart 3") . IncrementLef t 496# 

ActiveSheet . Shapes ( "Chart 3" ) . IncrementTop -8# 

ActiveSheet . Shapes ( "Chart 2") . IncrementLeft 81# 

ActiveSheet . Shapes ( "Chart 2") .IncrementTop 387# 

ActiveSheet . Shapes ("Chart 1") .IncrementLeft 69# 

ActiveSheet. Shapes ("Chart 1" ). IncrementTop -8# 

End Sub 
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APPENDIX G: OPTIMIZATION FORMULATION AND CODE 



A. NPS FORMULATION 

For each trial i, GAMS is used to determine the closest point of intersection of the 
submarine and torpedo trajectories, if the intersection occurs before TMAX. Otherwise, 
it determines the range at TMAX. 



Problem: 


For the CPA parameters (Time, Range) for the weapon and the target. 


INDEX: 


i 


TRIAL i=l..I 


Data: 


Xsoi 


initial submarine x position for trial i 




Ysoi 


initial submarine y position for trial i 




Xto 


initial torpedo x position for trial i 




Yto 


initial torpedo y position for trial i 




Vsxi 


submarine velocity in the x direction for trial i 




Vsy; 


submarine velocity in the y direction for trial i 




Vtxi 


torpedo velocity in the x direction for trial i 




Vtyi 


torpedo velocity in the y direction for trial i 




Tmax 


maximum run time of torpedo in seconds 


Variables: 


Rhi 


CPA range for trial i 



Ti CPA time for trial i 



Formulation for trial i: 
min 
s.t 



Output: Rh; 

L 

B. GAMS CODE 

* DAVID SOLDOW 

* CPA OPTIMIZATION ROUTINE 

$Title: CPA optimization routine 

SETS 

I 71*65/ 



PARAMETERS 



Xso (I) 


/I 


2881.801, 


(EDITED 


FOR 


SPACE) 


65 


2116.524/ 


Yso (I) 


/I 


-4971.37, 


(EDITED 


FOR 


SPACE) 


65 


437.9259/ 


Vsx(I) 


/I 


-5.16831, 


(EDITED 


FOR 


SPACE) 


65 


-1.31678/ 


Vsy (I) 


/I 


-1.88111, 


(EDITED 


FOR 


SPACE) 


65 


-3.61782/ 


Vtx ( I ) 


/I 


11.33185, 


(EDITED 


FOR 


SPACE) 


65 


29.35145/ 


Vty(I) 


71 


-28.0473, 


(EDITED 


FOR 


SPACE) 


65 


7.318137/; 



SCALAR TMAX /1145.5/; 
SCALAR Xto 70/; 



Rh; 

Rhi=sqrt(((Xso+Vsx*Ti )-(Xto+Vtx*Ti » A 2+ 
((Yso+Vsy* Ti )-(Yto+Vty* Ti » A 2) 
0<=Ti<+Tmax 
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SCALAR Xto /0/; 

SCALAR Yto /0/; 

Scalars 

Xsoi, Vsxi, Vtxi, Ysoi, Vsyi, Vtyi ; 

VARIABLES 

Z CPA RANGE FOR TRIAL I 

T CPA TIME FOR TRIAL I ; 

T.LO=0; 

T . UP=TMAX; 

EQUATIONS 

CPA OBJECTIVE VALUE FOR TRIAL i; 

CPA .. Z=e=SQRT (sqr ( (XsoI+VsxI*T) - (Xto+VtxI*T) ) 
+sqr ( (YsoI+VsyI*T) - (Yto+VtyI*T) ) ) ; 
MODEL SHOTCPA /ALL/; 

FILE REPORT/ SHOTCPA. RPT/ ; 

PUT REPORT; REPORT. ND=5; 

PUT /'TRIAL RH (I ) T(I)'; 

PUT / ' ’ ; 

LOOP (I, 

Xsoi=xso (i) ; 

Vsxi=vsx (i ) ; 

Vtxi=Vtx (i) ; 

Ysoi=Yso (i) ; 

Vsyi=vsy (i) ; 

Vtyi=vty (i) ; 

SOLVE SHOTCPA USING NLP MINIMIZING Z; 

PUT I.tl; 

PUT Z.l; 

PUT T. 1; 

) ; 

PUTCLOSE REPORT; 



C. OUTPUT 



TRIAL 


RH (I ) 


T (I ) 


1 


214.395 


186 


2 


214.5673 


124 


{EDITED 


FOR SPACE) 




65 


298.5145 


66 
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